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1.  ABSTRACT 


Computer  games  have  the  power  to  attract  and  hold  children’s  attention  for  extended  periods  of  time, 
often  exceeding  that  of  the  classroom  environment.  Intific’s  OurSpace  program  has  been  designed  and 
developed  to  offer  an  innovative  game  system  tailored  to  spark  the  interest  of  Pre-K  to  Grade  3  students 
in  scientific  concepts  and  lay  the  foundation  to  master  STEM  learning.  The  game  environment  is  hosted 
on  the  Internet  and  allows  various  levels  of  educational  interaction  and  play  with  graphics  and  related 
concepts  of  Social  Emotional  Learning  (SEE),  to  interest  and  inform  students  while  collecting  metrics 
that  can  be  used  by  educators  to  assess  the  performance  of  each  individual  student. 

DEVELOPMENTAL  APPROACH:  Our  plan  is  ambitious:  to  create  a  completely  new  way  of  using  the 
tools  of  our  modern  technological  culture  to  change  how  children  feel  about  being  educated,  how  teachers 
join  in  their  students’  journey,  how  our  society  engages  in  directing  that  process,  and  how  learning 
becomes  an  integrated  whole  that  moves  along  the  same  path  that  our  growing  global  technical 
interdependence  compels  us  to  travel.  To  accomplish  this  Intific  created  a  future-facing  space  theme, 
designed  a  platform  for  hosting  games  called  OurSpace,  and  designed  both  a  prototype  and  a  final  version 
of  one  game  called  RoboBall  from  a  suite  of  potential  STEM  topics,  continually  improving,  focusing, 
refining,  and  testing  that  game  throughout  Phase  II.  Intific’s  initial  OurSpace  game  RoboBall,  a  proof-of- 
concept  game  exemplar  that  focuses  on  Physical  Science  and  Science  as  Inquiry  involving  an  interactive 
understanding  of  force,  motion,  and  gravity.  OurSpace  serves  as  the  educational  Portal  that  the  RoboBall 
game  is  hosted  from.  It  allows  access  to  the  game,  the  data  and  the  editor  and  is  designed  for  future 
crowdsourced  refinement  and  so  that  additional  learning  games  can  be  easily  hosted. 

CONCEPTUAL  APPROACH:  Intific’s  OurSpace  program  is  designed  to  be  a  platform  that  fuses  of 
videogame  and  next-generation  learning  systems  to  address  our  emerging  needs:  it  is  a  platform  using 
games  that  create  a  multi-student  online  learning  environment  that  can  be  populated  with  an  ever- 
expanding  variety  of  learning  experiences.  OurSpace  is  also  based  on  a  novel  toolkit  approach  that  lets 
students  create  their  own  future  by  having  learning  tailored  to  their  chosen  field  through  ART  (Alternate 
Reality  Teaching).  In  such  a  game  and  learning  environment  every  student  is  given,  initially  through  the 
DARPA  ENGAGE  program’s  games,  the  key  visualizing  building  blocks  of  thought  and  learning  (Visual 
Thinking,  Amheim,  1969),  and  as  students  advance  -  ‘leveling  up’  in  the  parlance  of  the  game  industry  - 
they  tailor  learning  to  their  desires.  Playing  together  in  their  own  virtual  communities,  students  could 
sharpen  their  critical  thinking  and  logical  reasoning  skills  as  they  strive  to  ‘win’  entertaining  videogames 
with  deeply  imbued  learning  and  tailored  to  their  specific  educational  needs.  Using  these  tools,  students 
will  also  learn  to  share,  emphasize,  and  cooperate  with  each  other  to  win  and  succeed,  improving  peer 
relations,  lessening  bullying  and  laying  a  foundation  for  Social  Emotional  Learning  (SEE).  The  overall 
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process  used  in  OurSpace  we  call  Alternate  Reality  Teaching  (ART),  the  logical  extension  of  highly 
acclaimed  Alternate  Reality  Games  (ARGs)  of  video  gaming,  and  we  believe  that  this  unique  approach 
aligns  exceptionally  well  with  the  Standards -based  Education  approach  of  this  program. 

THEMATIC  APPROACH:  The  OurSpace  program  is  designed  to  use  a  space  theme  to  capture  the 
imagination  of  students  of  all  ages,  while  embracing  flexibly  by  adding  modern  educational  research  on 
learning  strategies,  gender  and  gaming,  and  social  emotional  learning.  The  OurSpace  program  provides  us 
with  a  unique  opportunity  to  make  curiosity  a  beacon  for  future  generations  by  creating  a  familiar  yet 
very  new  online  franchise  vision  that  connects  emotionally,  socially  and  intellectually  with  this  growing 
student  audience  and  realizes  that  vision  using  the  most  qualified  development  team  possible.  This 
approach  combines  both  the  space  exploration  theme  of  many  successful  entertainment  properties  and 
leverages  technologies  for  interconnectivity  and  communication  to  create  a  new,  shared  learning  space 
that  becomes  OURS,  as  opposed  to  MINE.  These  elements,  taken  together,  create  an  entertainment 
spectrum  for  gaming  that  evolves  in  sophistication  as  students  advance  in  age,  so  that  learning  is  additive 
and  integrated  as  opposed  to  siloed  and  restrictive  as  the  current  learning  approaches  provide. 

As  veteran  software  developers,  has  been  is  gifted  with  powerful  new  tools  with  which  to  ply  our  trade. 
Computers,  the  Internet,  Social  Media,  Databases,  Mobile  Devices,  Portable  Tools  and  an  astounding 
array  of  technological  innovations  now  cover  our  workbench.  Our  plan  is  ambitious:  to  create  a 
completely  new  way  of  using  the  tools  of  our  modern  technology  culture  to  change  how  children  feel 
about  being  educated,  how  teachers  join  in  their  journey,  how  our  society  engages  in  directing  that 
process,  and  in  how  learning  becomes  an  integrated  whole  that  moves  along  the  same  path  that  our 
growing  global  interdependence  compels  us  to  travel. 

Intific  believes  and  our  initial  (and  future)  test  results  will  demonistrate  that  in  the  modern  digital  age  it  is 
now  possible  to  fuse  this  method  of  self-paced  learning  with  classroom  processes  and  guidance  to 
energize  students  in  achieving  greater  levels  of  interest  and  performance  in  Science,  Technology, 
Engineering  and  Mathematics  (STEM)  topics.  The  cumulative  nature  of  computer  game  progression  ties 
extremely  well  the  educational  layering  required  to  teach  STEM  and  even  SEE  related  material. 

Our  research  and  development  in  this  final  phase  of  OurSpace  also  made  clear  to  us  that  our  approach  to 
educational  training  benefits  from  being  configurable  or  modifiable  by  the  educators  in  order  to  adapt  the 
material  to  the  learning  styles,  capabilities  and  ultimately  the  disabilities  of  individual  students.  In  this 
program  the  feedback  obtained  through  collected  data,  metrics  and  analytical  tools  have  assisted  our 
educators  and  teams  in  rapidly  identifying  the  changes  needed  for  each  individual  or  for  the  group  as  a 
whole.  Initial  testing  also  indicated  that  there  will  be  occasions  when  additional  educational  levels  need  to 
be  added  in  order  to  effectively  convey  and  expand  upon  the  original  game  approaches. 


2.  SUMMARY  OF  THE  OVERALL  PROGRAM  EFFORT 


The  initial  OurSpace  game  created  by  Intific  is  called  RoboBall  and  it  is  hosted  on  a  server  accessible  by 
any  student,  using  a  web  browser  from  a  mobile  device  with  Internet  access,  and  the  overall  system  was 
developed  according  to  the  following  phases,  with  this  summary  of  2014  concluding  efforts. 
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Detailed  Milestone  Schedule 


Closed  Alpha 


Game  Refinement  &  Testing 


Open  Beta 


Final  Game  Software 
(executable  &  code) 
Classroom  Transition  Plan 


PHASE  I:  The  overall  development  objective  of  the  program  in  this  phase  was  to  have  a  functional 
prototype  in  the  first  8  months,  and  this  was  achieved  through  cooperation  amongst  the  Intific  Team 
(University  of  Denver,  Texas  A&M  University,  and  Digital  Steamworks)  as  well  as  extensive 
collaboration  with  CRESST/UCLA  and  the  DARPA  staff.  We  would  like  to  recognize  all  these  groups 
for  their  assistance  and  invaluable  contributions  which  are  summarized  below. 


PHASE  H:  In  this  phase  Intific  focused  on  refining  the  ENGAGE  concept  while  incorporating  DARPA’ s 
feedback  into  the  ongoing  improvements  that  were  made.  This  included  building  a  Test-Ready  game  for 
CRESST  at  UCLA  to  use  to  evaluate  the  effectiveness  of  gameplay  and  learning  progression,  and 
working  with  BrainPOP  and  Sesame  Street  in  receiving  feedback  to  develop  an  effective  classroom 
transition  plan.  This  was  alll  done  by  using  the  in-game  assessment  data  generated  from  RoboBalL  The 
data  was  accessible  from  the  OurSpace  website  where  RoboBall  is  hosted.  Researchers  could  view  up  to 
100  data  logs  on  a  single  webpage  or  download  a  .csv  file  for  further  examination.  The  game  captured 
every  piece  of  data  from  gameplay,  with  the  information  most  frequently  used  by  research  partners  made 
readily  accessible  to  make  reviewing  test  results  more  expedient. 


2.1.  PHASE  II  LOCAL  AREAS 


Intific  worked  closely  with  our  CRESST/UCLA,  BrainPOP,  and  Sesame  Street  partners  to  refine  the 
prototype  design  during  Phase  II.  Key  Phase  II  Local  Areas  included: 

•  TA-2  Collaboration  to  identify  Physics,  Prediction  and  Social  Emotional  Learning  designs. 

•  Build  a  Test-Ready  game  to  be  used  for  focus  testing. 

•  Provide  data  from  gameplay  to  aid  in  assessment. 

•  Refine  the  editor  and  tools  used  to  create  RobobalL 

•  Collaborate  with  Sesame  Street  Workshop  to  leverage  their  insight. 

•  Transition  our  game  to  BrainPOP  to  facilitate  the  collection  of ‘‘big  data”. 
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2014  ENGAGE  Timeline 

External  Party 

1  Jan  Feb  Mar  Apr  May  June  | 

DARPA 

1.  Monthly  Report 

1.  Monthly  Report 

1.  Monthly  Report 

2.  Classroom  Transition  Plan 

1.  Monthly  Report 

2.  Updated  GDD/TDD 

1.  Monthly  Report 

2.  Phase  2  Final  Report 

Project 

Complete 

BrainPOP 

1.  Finalize  agreement 

2.  Design  changes 

1.  Start  integration  to  BP 
Jr.  site 

2.  Provide  BP  with  "game 
assets" 

1.  Integration  Complete 

1.  Transition  Support 

1.  Continue  Support  if 
needed 

CRESST 

1.  Review  Test  Feedback 

2.  Discuss  remaining 
testing 

1.  Final  Release  of 

RoboBall 

2.  Begin  Writing 

Classroom  Transition  Plan 

1.  Complete  Classroom 
Transition  Plan 

1.  Begin  writing  the  Final 
Report 

1.  Complete  Phase  2 

Final  Report 

Sesame  St. 

1.  Send  documentation 
and  game  link 

1.  Review  RoboBall 

2.  Provide  initial  feedback 

1.  Supply  feedback  for  the 
Classroom  Transition  Plan 

1.  Provide  input  for  the 
Final  Report 

1.  Provide  RoboBall 

review  documentation 

2.2.  KEY  PROGRAM  MILESTONES 

The  OurSpace  platform  and  the  RoboBall  game  were  developed  according  to  the  following  major 
milestones.  Key  activities  throughout  this  program’s  full  period  of  performance  were: 

•  Prototype  Game  June  1, 2012  Complete 

o  Flash  Version:  Deployed  on  Intific  public  server  available  to  informed  audiences 
o  Support  for  CRESST/UCLA  preliminary  testing 

•  Test  Ready  Game  Nov.  27, 2012  Complete 

o  HTML-5  version,  programmer  toolset,  requires  game  programmer 
o  CRESST/UCLA  individual  trained 

•  Test  Ready  Toolset  April  1, 2013  Complete 

o  More  robust  toolset,  more  contributor  feedback 

•  Final  Version  Game  April  30, 2014  Complete 

o  Complete  toolset,  metrics,  and  controls 
o  Crowd  source-capable  game  ready  for  deployment 


3.  WORK  ACCOMPLISHED  IN  PHASE  I  (BASE  PERIOD) 


In  broad  summary,  during  Phase  I  Intific  built  a  prototype  game  called  RolloBall  for  the  OurSpace 
platform,  and  in  Phase  II  we  extensively  refined  that  game.  We  did  this  by  achieving  quarterly  milestones 
according  to  the  Phase  I  tasks  below  as  the  foundation  for  all  Phase  II  activity. 

3.1.  TASK  1:  IDENTIFY  DESIRED  EDUCATIONAL  OUTCOMES,  STEM  FOCUS, 
DEVELOP  GAME  CONCEPT(S)  AND  STORYBOARD(S) 

Detailed  Task  Description  Complete 

Our  team,  with  input  from  CRESST  (UCLA)  and  DARPA,  developed  storyboards  and  a  list  of 
learning  objectives  and  other  game  function  requirements  for  OurSpace.  OurSpace  is  a  story  about  a 
team  of  characters  who  must  journey  from  their  ecologically  distressed  world  to  three  different 
planets  in  an  effort  to  learn  skills  to  repair  their  home.  In  the  process,  they  end  up  helping  the 
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residents  of  each  planet.  The  design  provided  for  differing  learning  objectives  on  each  world  with 
incremental  levels  teaching  differing  STEM  concepts  in  a  progressive  manner. 

Deliverables: 

Storyboard  document:  Submitted  2  Nov  2011,  and  Approved 
Game  Design  Document:  Submitted  27  Nov  2011  and  Approved 

3.2.  TASK  2:  BUILD  GAME,  GAME  ARCHITECTURE,  &  COMPLETE 
DEVELOPMENT  TESTING. 

Detailed  Task  Description  Complete 

Our  contracting  group.  Digital  SteamWorks,  quickly  developed  a  Elash-based  prototype  of  the  game, 
implementing  a  total  of  32  levels  spread  across  the  three  worlds. 

Intific  concurrently  developed  the  OurSpace  web  site  and  with  features  for  capturing  and  reporting 
individual  performance  measures.  Typical  information  captured  involves  mouse  clicks,  time 
measurements  and  achievements  reached.  Collectively  we  integrated  this  system  and  began  hosting 
that  environment  and  application  on  a  Web  Server  in  that  May.  CRESST/UCLA  then  started 
preliminary  testing. 

Denver  University,  with  support  from  Intific,  developed  “OurSpace  Adaptation:  A  White  Paper” 
(included  with  this  report)  presenting  concepts  for  analysis  in  Phase  II.  Adaptation  may  encompass 
many  areas,  so  the  team  will  evaluate  these  concepts,  correlated  with  the  prototype  game  metrics  to 
determine  what  types  of  adaptation  could  be  effectively  implemented  in  subsequent  development.  The 
primary  areas  we  are  investigating  are  adjustment  of  game  content  for  students  with  sensory 
disorders,  adjustment  of  game  teaching  style  based  on  learning  styles  of  individuals,  adjustment  of 
game  content  and  pacing  through  a  teacher  console,  and  real-time  automatic  game  content  adjustment 
based  on  player  performance. 

Texas  A&M  University  developed  concepts  for  further  Social/Emotional  Learning  (SEE)  objectives 
that  could  be  incorporated  into  Phase  2  efforts,  their  analysis  being  performed  on  the  Prototype  Game. 
This  included  an  investigation  into  how  to  successfully  measure  SEE  through  measures/metrics  rather 
than  in  person  through  observation  and  interviews,  which  is  highly  beneficial  in  a  game  that  is  to  be 
widely  distributed. 

One  of  the  concepts  in  the  early  DARPA  project  discussions  was  Crowd  Sourcing.  While  this 
concept  is  still  being  refined  with  BrainPOP,  when  coupled  with  the  concepts  of  Adaptation,  it 
became  obvious  that  a  customizable  environment  needed  to  be  created  to  support  the  further 
objectives.  Intific  developed  a  “Game  Creation  Framework”  which  will  be  used  to  support  the  future 
objective  and  allow  OurSpace  to  evolve.  In  this  task  Intific  successfully  transformed  3  key  levels  of 
the  prototype  game  into  this  platform  and  will  further  develop  the  environment  and  convert  the 
OurSpace  game  into  this  framework  in  order  to  support  the  Phase  2  Test  Ready  Game  Objective. 

Deliverables: 

Alpha  game  software:  Submitted  28  Eeb  2012 
Beta  game  software:  Submitted  27  Apr  2012 
Prototype  game  software:  Submitted  with  this  report 
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Source  Code:  The  source  code  for  the  Prototype  Flash  based  program  was  sent  on  a  DVD  via 
FedEx.  The  source  code  and  configuration  instructions  for  the  Web  Server  which  is  built  upon 
commercial  products  is  not  included  on  this  DVD*. 

Executable:  The  executable  for  the  program  is  hosted  upon  a  web  server.  Intific  has  hosted  this  on  a 
public  space  which  is  accessible  to  DARPA/ONR  and  other  teams.  We  will  continue  to  host  in  this 
space  during  Phase  II  so  that  evolutions  are  visible  to  the  entire  team  upon  release.  The  location  of 
this  portal  is  http://engagedemo.intific.com/.  The  user  may  register  at  this  location  and  create  an 
account  for  their  use.  Once  a  user  has  logged  in  the  initial  attempt  to  play  the  game  will  require  two 
additional  credentials.  These  credentials  will  be  emailed  to  DARPA  under  separate  cover. 

^Intific  will  provide  the  list  of  third  party  products  and  develop  a  configuration  instruction  for  the 
government  if  required.  Since  this  is  a  rapidly  executing  program  with  the  next  delivery  in  Phase 
2  scheduled  for  November,  and  the  Prototype  Game  will  be  replaced  with  a  more  functional  game 
engine,  we  have  not  now  spent  the  time  preparing  instructions  and  process  for  external  parties. 

3.3.  TASK  3:  PROGRAM  MANAGEMENT 


Detailed  Task  Description  Complete 

Intific  has  coordinated  the  efforts  of  our  team  with  the  overall  objectives  of  the  program  and  the 
periodic  DARPA-led  meetings.  The  development  of  the  initial  game  was  completed  earlier  than 
originally  forecast,  allowing  us  to  focus  on  the  quality  of  the  experience  and  the  depth  of  the  metrics 
measured. 

Deliverables: 

No  Physical  Deliverables 

3.4.  TASK  4:  MEETINGS,  DOCUMENTATION  &  REPORTING 


Detailed  Task  Description  Complete 

Team  participation  in  the  DARPA  meetings  yielded  multiple  benefits  to  our  development  program. 
Our  documentation  has  been  delivered  and  accepted  along  with  our  reports. 

Deliverables: 


Slide  Presentations:  Submitted  17  Aug  2011 
Monthly  Reports:  Submitted  as  required 
Final  Phase  Report:  The  Phase  I  Report  document 

Implementation  Documentation:  Due  1  Month  after  Phase  1  complete  (20  Jun  2012) 
Technical  Papers:  Provided. 


3.5.  PHASE  I  SAMPLE  DESIGN  VISUALIZATIONS 


In  demonstrating  the  changes  to  the 
RoboBall  game  undertaken  in  Phase 
II,  it  is  useful  to  conclude  a  Phase  I 
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summary  with  a  keyframes  of  how  the  initial  prototype  game  operated. 

This  prototype  game  had  thirty-two  (32)  levels  spread  across  three  Worlds.  A  Progression  chart 
served  as  the  master  list  of  the  levels,  which  was  created  by  merging  the  learning  progression  with  the 
game  skill  progression  in  a  manner  that  served  the  goals  of  each.  The  result  was  a  game  that 
introduced  game  mechanics  in  parallel  with  learning  targets.  These  screens  below  show  Level  1  from 
both  Worlds  1  and  2. 


Game  World  1 


Game  World  2 


4.  WORK  ACCOMPLISHED  IN  PHASE  II 


Guided  by  the  ENGAGE  goals  of  facilitating  young  student  interest  in  Science,  Technology,  Engineering 
and  Mathematics  (STEM)  topics  through  involvement  in  education  game  playing,  in  Phase  II  Intific 
worked  with  CRESST/UCLA,  Sesame  Street  Workshop,  and  BrainPOP  to  fully  develop  a  revised  version 
of  our  physics-based  educational  game.  The  specific  Task  elements  in  Phase  II  were: 

4.1.  TASK  I:  REPINE  THE  ENGAGE  GAME  CONCEPT 

•  Primary:  Intific,  Inc. 

•  Supporting:  Intific,  Inc.,  Texas  A&M,  and  the  University  of  Denver 

Detailed  Task  Description  Complete 

In  this  task,  the  Intific-lead  team  will  reevaluate  the  game  concept  to  be  executed  in  Phase  2.  The 
team  will  integrate  feedback  from  DARPA,  educational  experts  on  our  team,  and  other  members  of 
the  ENGAGE  TA-2  community.  As  part  of  the  refinement  process,  the  game  concept  will  be  aligned 
with  customers  (students)  as  identified  by  the  TA-2  ENGAGE  participants. 

Deliverables: 

•  Revised  game  design  document,  as  part  of  the  living  documentation  of  the  software 
development. 

Exit  Criteria:  Submission  of  documents  and  DARPA  approval  of  documents. 

University  Contribution:  Both  Texas  A&M  and  University  of  Denver  played  significant  roles  in 
evaluation  of  the  pedagogical  aspect  of  the  game  and  game  play. 
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4.2.  TASK  2:  BUILD  TEST-READY  GAME 

•  Primary:  Intific,  Inc 

•  Supporting:  University  of  Denver 

Detailed  Task  Description  Complete 

Intific  will  be  responsible  for  the  creation  of  a  test-ready  game  for  the  ENGAGE  program.  This  will 
involve  the  production  of  games  as  described  in  Phase  2  of  the  technical  plan  within  this  proposal  and 
influenced  by  the  activities  of  Phase  I.  The  preparation  for  deployment  and  use  within  the  TA-2  area 
will  involve  a  rigorous  Q&A  process  to  ensure  that  the  game  is  playable,  major  bug-free,  and  useable 
by  children  of  the  appropriate  age  category. 

Deliverables: 

•  Test-ready  game  software,  6  months  after  Phase  2  contract  award. 

Exit  Criteria:  Submission  of  software  executables,  and  DARPA  receipt  of  software. 

4.3.  TASK  3:  EUNCTIONAL  TESTING,  REEINEMENT  OE  GAME  &  EINAL  GAME 
DEVELOPMENT 

•  Primary:  Intific,  Inc 

•  Supporting:  University  of  Denver 

Detailed  Task  Description  Complete 

Based  on  input  from  and  interactions  with  TA-2  providers  and  students  using  the  game,  Intific,  Inc. 
will  augment,  edit,  or  revise  the  game  during  the  main  testing  period.  These  are  assumed  to  be 
quarterly  updates,  allowing  time  for  the  development  team  to  respond  to  needed  changes. 

Deliverables: 

•  Revised  game  software:  delivered  as  either  updates  or  new  executables,  quarterly  following 
the  initial  release  of  the  test-ready  game  software. 

•  Pinal  Game  software:  delivered  to  DARPA  at  23  months  after  Phase  2  contract  award. 

Exit  Criteria:  Submission  of  software  executables,  and  DARPA  receipt  of  software 

4.4.  TASK  4:  INTERACTION  WITH  TA-2  PROVIDERS 

•  Primary:  Intific,  Inc. 

•  Supporting:  University  of  Denver 

Detailed  Task  Description  Complete 

This  task  is  primarily  a  coordination  task,  but  involves  interaction  with  the  ENGAGE  TA-2 
performers.  As  TA-2  will  be  responsible  for  the  test  plan  and  overall  deployment  of  the  game 
software,  Intific,  Inc.  will  collect  inputs  and  distribute  them  to  the  developer  and  various 
educational  partners.  University  of  Denver  will  assist  with  communication  to  the  TA-2  team. 

Deliverables: 

•  Input  to  be  used  by  Intific,  Inc.  in  the  execution  of  Phase  2,  no  formal  deliverables. 

Exit  Criteria:  Occurrence  of  interaction,  likely  via  teleconference. 
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University  Contribution:  University  of  Denver  will  assist  with  communication  to  the  TA-2 
performers. 

4.5.  TASK  5:  PROGRAM  MANAGEMENT  -  INTERNAL  &  EXTERNAL  PROGRAM 
COORDINATION 

•  Primary:  Intific,  Inc. 

•  Supporting:  Texas  A&M  University 

Detailed  Task  Description  Complete 

This  task  is  the  program  management  and  coordination  task.  This  will  be  an  Intific,  Inc.  lead  task, 
since  Intific  will  be  responsible  for  pulling  together  all  of  the  various  subcontracting  entities  and  TA- 
2  performers.  Intific  will  also  serve  as  the  main  interface  to  DARPA  through  the  management  roles 
assigned  to  this  effort.  Supporting  Intific  in  this  role  is  Texas  A&M  University,  who  will  assist  with 
coordination  of  the  various  university  entities  in  the  effort. 

Deliverables: 

•  No  physical  deliverables.  Outcome  is  team  coordination  as  described  in  the  Management  Plan 
section  of  this  proposal. 

Exit  Criteria:  Occurrence  of  meetings  and  teleconferences  as  planned. 

4.6.  TASK  6:  DARPA  PROGRESS  MEETINGS,  DOCUMENTATION  &  REPORTING 

•  Primary:  Intific,  Inc. 

•  Supporting:  Texas  A&M  University,  University  of  Denver 

Detailed  Task  Description  Complete 

This  task  supports  the  various  meetings,  reports  and  documentation  required  as  deliverables  for  the 
DARPA  ENGAGE  program.  In  the  role  of  integrator,  Intific,  Inc.  will  take  primary  responsibility  for 
the  presentations  and  reports.  Intific  will  be  supported  by  Intific,  Texas  A&M  University,  and 
University  of  Denver  in  the  production  of  all  data,  screen  shots,  and  text  required  to  complete  the 
items.  The  publication  of  technical  papers  will  be  led  by  Texas  A&M  University  and  University  of 
Denver  through  their  respective  academic  institutions. 

Deliverables: 

•  Slide  Presentations:  1  month  after  Kickoff  meeting,  1  month  after  annual  review 

•  Monthly  Progress  Reports:  Monthly,  starting  30  days  after  Phase  2  contract  award 

•  Final  Phase  2  Report:  24  months  after  contract  award 

•  Implementation  Documentation:  1  month  after  end  of  Phase  2 

•  Technical  Papers:  Written  and  published  as  appropriate  during  the  effort 

Exit  Criteria:  Submission  of  documents  to  DARPA  with  DARPA  receipt  of  materials. 


5.  PHASE  II,  TASK  I  -  NEW  DESIGN  AND  DEVELOPMENT  DIRECTIONS 


5.1.  WHAT  IS  ROB  OB  ALL? 

In  RoboBall,  students  learn  both  physics  and  social  emotional  learning  concepts  through  problem 
solving.  The  goal  of  the  game  is  to  make  the  RoboBall  reach  a  treasure  chest  while  avoiding 
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colliding  with  an  asteroid.  Students  manipulate  different  variables  (force  direction  and  magnitude, 
mass)  to  control  the  motion  of  RoboBall. 


5.2.  SUMMARY  OF  GAME  REFINEMENT  RECOMMENDATIONS  FOR  TASK  1 

In  Phase  II  Intific’s  educational  partners  provided  a  great  deal  of  input  and  insight  into  game 
refinements  to  best  align  the  overall  experience  with  the  required  STEM  and  SEE  learning 
objectives.  The  above  section  highlights  the  visual  and  play  changes  as  implemented  because  of 
these  recommendations.  The  following  sections  call  out  those  recommendations  in  more  detail, 
and  how  Intific  subsequently  modified  the  overall  RoboBall  game  in  Phase  II. 
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CRESST's  Feedback 

Solution 

1. Problematic  introduction  to  concepts  (sequence  and 
confounding  variables). 

1. The  September  release  will  have  demonstration  levels  that  communicate  the  concepts 
covered  In  the  upcoming  levels. 

2. Creatlon  of  "units"  that  build  on  a  speciTx:  game  mechanic  and  learning  target. 

2. Knowledge  of  relevant  physics  In  not  useful  in  solving 
puzzles. 

1.  We  have  built  scaffolding  that  builds  up  the  learning  progression  and  game  mechanic 

Intuition. 

3. Presence  or  Absence  of  gravity  Is  not  explicitly 
communicated. 

1.  There  will  be  storyboard  and  In-game  art  that  communicates  that  absence  of  gravity. 
This  will  Include  a  world  In  which  all  levels  are  played  In  space,  so  the  art  communicates 
that  the  gravity  Is  nqghglble. 

4.More  Icons  are  needed  tor  the  relevant  physics 
concepts. 

1.  There  will  be  graphical  aids  for  vlsuafizlng  nuss,  gravity,  and  distance. 

5. Achievements  should  be  more  closely  tied  to  learning 
goals  and  objectives  of  the  game. 

1.  With  the  adaptive  learning  levels,  every  level  Is  tagged  with  the  specific  learning  goal  It 
corresponds  to.  This  will  allow  us  to  give  nx>re  accurate  achievements  based  on  the 
learning  progression. 

6.1  n  the  Achievement  viewer.  Social  Emotional  Learning 
Is  referenced  as  an  acronym;  SEL  Young  children  will 
not  understand  this. 

1.  With  the  Introduction  of  learning  based  achievements,  this  will  no  longer  be  an  Issue. 

2.  The  reference  will  be  replaced  with  another  term. 

Z.Character  use  or  appearance  frequency  nuy  favor 
certain  characters. 

1.  The  character  use  has  been  evaluated  and  we  will  adjust  when  and  where  the 
characters  are  shown. 

S.Soclal  Emotional  Learning  scenarios  designs  nuy 
need  another  revision. 

1.  In  general,  the  development  of  the  SEL  scenarios  takes  3  weeks  per  scenario,  we  have 
worked  with  CRESST  to  design  each  scenario  and  show  Iterative  progress  throughout 
the  creation  process. 

3.  We  will  revisit  the  tuning  of  the  scenarios  after  Sept.  30th. 

Sesame's  Feedback 

Solution 

l.There  are  no  stakes  for  losing  or  Incorrectly 
answering. 

1.  There  will  be  a  system  Implemented  that  tracks  the  #  of  attempts  In  a  given  unit. 

2.  The  player  will  have  a  chance  to  use  each  crew  member  to  complete  the  level.  If  the 
player  try's  and  falls,  the  art  will  Indicate  that  they  have  one  less  attempt  left  by  showing 
the  character  greyed-out. 

2. Each  world  should  have  consistent  properties,  so  as 
the  player  moves  through  the  game  they  can  use  their 
past  experience  and  knowledge  to  solve  future  puzzles. 

1.  Moving  forward  each  world  will  have  specific  "environmental"  properties,  ex.,  gravity 
or  n^figlble  gravity. 

3. The  Introduction  of  new  elements  need  scaffolding. 

1. The  September  release  will  have  denrkonstratlon  levels  that  communicate  the  concepts 
covered  In  the  upcoming  levels. 

2. Creatlon  of  "units"  that  build  on  a  specific  game  mechanic  and  learning  target. 

4.lt  would  be  nice  to  have  voice  over  for  tutorials. 

1.  One  of  the  restrictions  for  this  project  was  the  classroom  environment.  Due  to  the 
noise  restrictions  and  the  use  of  headphones  In  the  classroom,  we  decided  to 
deprlorltlze  the  production  of  sound. 

2.  We  can  revisit  this  issue  after  Sept.  30th. 

5. The  story  line  seems  a  bit  disjointed  from  the  rest  of 
the  experience. 

1.  The  storyline  has  been  revisited. 

6.The  tutorials  should  be  Interactive. 

1.  We  are  working  to  nuke  the  tutorials  Interactive  for  Sept.  30th. 

2.  Players  will  be  able  to  Interrupt  the  tutorial  video  by  chcklng  In  the  frame. 

Note  that  BrainPOP  also  provided  extensive  input  on  the  RoboBall  game,  and  that  feedback  is 
included  at  the  end  of  this  report  as  it  pertains  to  next-step  activities  involving  this  program. 

5.3.  GAME  REFINEMENT  AND  TECHNICAL  DESIGN  OVERVIEW 

The  purpose  of  the  following  material  is  to  familiarize  the  reader  with  the  design  changes  made  to 
RoboBall  throughout  Phase  2,  based  upon  the  feedback  summarized  above.  This  material  illustrates 
and  explains  the  game  designs,  mechanics  and  technologies  used  to  create  the  most  recent  version  of 
the  game. 
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This  portion  of  this  Final  Report  covers  the  “why’s  and  how’s”  of  the  creative  and  technical  design. 

5.4.  STORY  REDESIGN  PROBLEMS 

This  section  describes  changes  made  to  the  story  in  RoboBall  and  explains  revisions  made  to 
alleviate  problem  areas.  RoboBall’s  initial  story  suffered  from  the  following  problems: 

•  Story  narrative  was  too  involved  to  communicate  in  the  stand-alone  game 

•  Story  elements  did  not  match  physics  game  play  elements 

•  Story  elements  did  not  match  SEE  game  play  elements 

Given  that  RoboBall  will  function  as  a  standalone  online  game,  the  team  decided  to  make  the  story 
self-contained  within  the  digital  experience  so  that  it  does  not  require  peripheral  storybooks  or  similar 
devices.  Additionally,  previous  efforts  to  unite  game  play  and  story  have  introduced  extraneous  game 
play  elements  that  distract  from  the  physics  mechanics. 

Rather  than  add  elements  to  the  game  to  support  a  story  that  is  too  large  to  convey,  we  also  reduced 
the  scope  of  the  story  to  match  the  elements  already  present  in  the  game.  This  editorial  scoping  also 
serves  to  add  greater  emphasis  to  the  desired  elements. 

5.5.  STORY  REDESIGN  SOLUTIONS 

Based  upon  educator  feedback,  we  reduced  the  scope  of  RoboBalVs  storyline  in  the  following  ways: 

1.  Audited  existing  story  elements,  identified  those  which  correspond  to  existing  game  elements 

2.  Consolidation:  when  a  story  element  had  an  analog  in  the  game,  kept  it,  changing  any  visual 
or  functional  properties  to  match  the  existing  game  element 

3.  Streamlining:  when  a  story  element  had  no  analog  in  the  game,  removed  it. 

The  revised  storyline: 

A  huge  meteor  has  collided  with  the  Queen's  "World  Ark,"  scattering  it's  cargo  through  space  and 
across  the  planets  of  her  home  system.  You  and  your  friends  must  journey  across  the  solar  system 
and  retrieve  her  peoples'  treasures. 

Consolidation: 

1.  "Gifts"  become  "Chests,"  which  match  the  treasure  chests  in  the  SEE. 

2.  SEE  levels  communicate  the  opening  of  chests  to  retrieve  the  collectibles  contained  inside. 

Streamlining: 

Abandon  idea  of  cleaning/restoring  the  planet 

5.6.  OUR  GAME  STORY  TELLING  APPROACH 

The  in-game  approach  to  cinematic  storytelling  results  in  the  following  graphic  novel  approach  to  all 
RoboBall  cinematics.  These  provide  easy-to-read  and  exciting  interludes  that  draw  students  into  the 
game  and  provide  context  for  all  of  their  game  and  player  interactions. 
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5.7.  OUR  SOCIAL  EMOTIONAL  LEARNING  (SEL)  APPROACH 

The  power  of  video  games  to  remove  the  individual  persona  and  create  the  compulsion  to  engage 
with  a  digital  system  may  be  powerful  allies  in  addressing  Social  Emotional  Learning.  In  Phase  1 
CRESST/UCLA  was  tasked  with  developing  the  storybook  to  accompany  the  OurSpace  system’s 
RoboBall  game  that  would  expand  the  SEL  storyline  under  educator-lead  guidance  appropriate  to  the 
educational  and  emotional  maturity  of  the  students. 

Unfortunately  CRESST/UCLA  was  unable  to  deliver  this  storybook.  To  keep  the  social  emotional 
learning  objectives  intact,  Intific  developed  storyboard  intro  movies. 

These  introductory  in-game  movies  gave  the  players  background  information  about  the  characters, 
and  also  set  up  the  social  emotional  levels  by  presenting  a  situation  in  which  they  needed  to  identify 
the  emotions  the  characters  were  feeling. 

Examples  of  this  are  shown  below. 
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Figure  4.0-  Social  Emotional  Level  4  Intro  Movie  -  Setting-up  the  social  situation 


Figure  4.1  -  Social  Emotional  Level  4  Decision  Screen  -  How  do  the  characters  feel? 

5.8.  PRIZES  AND  LIVES 

Student  engagement  is  vital  in  learning  games  and  there  were  several  recommendations  of  how  to 
enhance  it  and  create  greater  player  motivation,  as  summarized  below. 
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5.8.1.  ENGAGEMENT  AND  MOTIVATION  IN  ROBOBALL 

This  section  describes  the  "Prizes”  and  "Lives"  systems  in  RoboBall.  The  "Prizes"  system 
supplies  mid-term  engagement,  while  the  "Lives"  system  provides  motivation  to  solve  levels 
through  analysis  rather  than  trial  and  error.  Before  we  discuss  prizes  and  lives  in  detail,  we  first 
provide  an  overview  of  Engagement  and  Motivation  design  goals  for  RoboBall.  We  then  outline 
the  goals  and  requirements  for  the  Prizes  and  Lives  systems. 

Engagement  refers  to  the  game's  ability  to  hold  the  player's  attention  over  some  period  of  time. 
RoboBall  has  three  distinct  units  of  time: 

•  Short-term:  duration  of  a  single  level 

•  Mid-term:  duration  of  a  group  of  related  levels,  referred  to  as  a  "unit" 

•  Long-term:  duration  of  a  play-through  of  the  entire  game 

Gameplay  provides  short-term  engagement,  and  the  story  cut  scenes  provide  long-term 
engagement.  We’ve  also  added  the  accumulation  of  Prizes  to  supply  mid-term  engagement. 

Motivation  refers  specifically  to  the  player's  desire  to  solve  levels  by  thinking  them  through, 
rather  than  using  trial  and  error.  Previously,  we  had  no  motivational  mechanics;  so  many  students 
resorted  to  trial  and  error  to  solve  levels.  Now  we  are  using  limited  Lives  to  motivate  players  to 
understand  the  physics  principles  required  to  solve  each  level. 

5.8.2.  PRIZES  AND  ENGAGEMENT 

The  concept  of  leveling  up  is  prevalent  in  videogames  and  rewards  are  a  powerful  motivator  for 
ongoing  student  involvement.  A  Prize  System  was  also  developed  to  foster  mid-term  engagement 
as  follows: 

•  Each  time  a  user  successfully  solves  a  level,  the  game  awards  him/her  a  Treasure  Chest 

•  Chests  remain  closed  until  the  player  completes  the  last  level  in  the  current  Unit 

•  If  the  player  fails  a  Unit,  he  loses  these  chests  he  accumulated  during  that  Unit 

•  When  the  player  successfully  completes  the  last  level  in  the  Unit,  the  game  displays  a 
variant  of  the  "End  Level"  dialog  box  that  displays  an  open  chest  filled  with  question 
marks 

•  When  the  player  successfully  completes  the  last  level  in  the  Unit,  the  game  adds  stickers 
to  the  player's  "collectibles"  pages,  equal  in  number  to  the  chests  collected 

•  When  the  player  completes  the  game,  the  game  unlocks  all  remaining  collectibles 

5.8.3.  LIVES  AND  MOTIVATION 

Another  important  aspect  of  videogames  that  enhances  player  motivation  by  giving  them  ‘skin  in 
the  game’  is  the  concept  of  lives/attempts.  The  new  Lives  system  provides  motivation  as  follows: 

•  The  player  begins  each  Unit  with  3  "Lives,"  represented  by  the  3  crew  members 

•  Each  time  the  player  fails  a  level,  one  of  the  crew  members  "sits  out"  the  remaining  levels 

•  When  the  last  crewmember  "sits  out",  the  player  fails  the  current  Unit  and  must  start  over 
at  the  Unit's  first  level 
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5.9.  NEW  USER  INTEREACE  REQUIREMENTS 

In  these  redesign  and  enhancement  efforts  it  became  clear  that  RoboBall  required  several  UI  changes 
to  support  the  new  mechanics.  Specifically: 

•  The  'Orb  Meter'  has  become  the  Current  Player  Portrait 

o  The  portrait  displays  the  bust  of  the  current  crew  member 

o  When  the  current  crew  member  returns  to  the  ship,  the  portrait  displays  the  next  crew 
member,  and  so  on. 

•  The  "Level  Success"  screen  features  a  Treasure  Chest  instead  of  a  Gift  Box 

o  The  Chest  will  appear  closed 
o  The  'Rank'  text  has  changed  to  "Chests" 

o  The  "Chests"  text  will  display  the  number  of  chests  so  far  collected  in  the  Unit 

•  The  "Level  Eailed"  screen  features  the  3  crew  members  in  addition  to  the  Captain 

o  Initially,  the  three  crew  members  will  all  stand;  as  the  player  fails  levels,  each 
member,  in  turn,  will  sit  down 

•  After  the  player  completes  the  last  level  in  a  Unit,  the  game  displays  the  Unit  Success  dialog 
variant 

o  The  Unit  Success  dialog's  "Chests"  text  show  the  number  of  chests  collected 
throughout  the  Unit 

o  The  Unit  Success  dialog  displays  a  large,  open  chest  filled  with  question  marks 
o  If  the  player  returns  to  the  main  menu  after  seeing  the  Unit  Success  dialog,  he  or  she 

will  see  the  "New  Collectibles"  notification  (shaped  like  a  chest)  on  the  Collectibles 
button 

5.10.  ASSESSMENT  AND  ADAPTIVITY  DESIGN 

This  section  outlines  the  current  design  for  Assessment  and  Adaptivity  within  RoboBall.  As  we 
iterated  on  our  designs,  we  updated  this  content  to  reflect  the  current  state  of  the  project. 

RoboBall  uses  "pre-test"  and  "post-test"  assessment  levels  to  track  student  performance  as  a  function 
of  levels  played.  We  have  divided  the  game  into  "units"  -  groups  of  levels  that  address  a  specific 
physics  concept.  Most  units  start  with  "demo  levels"  that  introduce  new  game  mechanics.  After  the 
demo  levels,  students  play  a  "pre-test  assessment  level"  to  gauge  their  understanding  of  the  unit's 
physics  concept  BEEORE  PLAYING  ANY  OE  THE  INSTRUCTIONAL  CONTENT. 

Next  come  a  series  of  instructional  levels,  the  number  and  nature  of  which  change  based  on  the 
player's  performance  (see  'Summary:  Adaptivity'  below).  After  completing  the  instructional  levels, 
players  must  complete  a  "post-test  assessment"  which  tests  understanding  of  concepts  from  earlier 
units. 

5.11.  SUMMARY:  ADAPTIVITY 

RoboBall  employs  two  forms  of  adaptive  content  management. 

•  Eirst,  it  uses  a  "save-point"  system  common  to  many  video  games:  if  players  fail  three  times 
during  a  unit,  the  game  returns  to  the  start  of  the  unit. 
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•  Second,  it  tracks  player  performance  and  assigns  additional  instructional  levels  to  players 
scoring  less  than  80%  on  a  unit's  learning  targets. 

All  of  these  tracking  metrics  are  part  of  the  overall  evaluation  data  set  available  concerning  the 
performance  of  each  student. 

5.12.  REVISED  GAME  STRUCTURE 

In  enhancing  RoboBall  following  the  prototype  version  of  Phase  I,  a  great  deal  of  new  engineering 
was  required.  The  game  now  relies  on  the  following  new  game  structures  to  implement  our  new 
assessment  and  adaptivity  algorithms: 

•  Units  -  group  of  levels  designed  to  explore  a  specific  physics  concept. 

•  Demo  Levels  -  introduce  new  game  mechanics.  Has  only  one  configuration,  so  students  can't 
fail. 

•  Pre-test  Levels  -  assess  the  student's  understanding  of  new  physics  concepts  BEFORE  playing 
instructional  levels  that  address  the  concepts. 

•  Exercise  Levels  -  present  new  physics  concepts  in  a  variety  of  ways  to  appeal  to  different 
learning  styles. 

•  Adaptive  Levels  -  optional  content  provided  to  students  who  score  lower  than  80%  on  the 
Exercise  levels. 

•  Post-test  Levels  -  assess  the  student's  understanding  of  physics  concepts  covered  in  the  previous 
unit. 

•  Social/Emotional  Learning  (SEL)  Levels  -  assess  the  student's  understanding  of  social  cues  and 
interactions. 

5.13.  REVISED  GAME  FLOW 

In  order  to  accommodate  the  needs  of  Adaptivity  and  Assessment,  it  was  also  necessary  that  Intific 
the  RoboBall  game  flow  as  follows: 

1.  The  game  consists  of  "units"  of  levels,  with  each  level  within  a  unit  relating  to  the  unit's 
overall  learning  goal. 

2.  If  a  unit  involves  a  new  game  mechanic,  the  first  levels  in  the  unit  will  demonstrate  this 
mechanic  via  demo  levels. 

3.  Immediately  following  any  demo  levels,  the  user  will  play  an  assessment  level,  which 
measures  students'  understanding  of  learning  goals. 

4.  After  taking  the  initial  assessment,  students  will  play  two  or  more  exercise  levels,  designed  to 
illustrate  the  unit's  learning  goal. 

5.  After  completing  the  final  exercise,  the  game  will  assess  player  performance  and  assign 
additional  adaptive  levels  if  the  player's  score  falls  below  80%  on  relevant  learning  targets. 

6.  If  assigned  no  adaptive  levels,  or  once  finished  with  adaptive  levels,  the  player  will  play  a 
"post-test"  assessment  level. 

7.  After  completing  the  post-test  assessment,  the  player  must  complete  a  social/emotional 
learning  level  (SEL). 

8.  After  completing  all  units  in  the  current  world,  the  player  proceeds  to  the  next  world  (NOTE: 
the  Open  Beta  version  of  RoboBall  has  only  one  world). 
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9.  After  completing  all  worlds,  the  game  presents  a  "Celebration”  screen  which  then  returns  the 
player  to  the  main  menu. 

See  the  subsequent  Adaptive  Gameplay  Flow  section  for  a  visual  representation  of  this  new  flow. 

5.14.  REVISED  REWARD  SYSTEM 

Intific’s  new  reward  system  for  RoboBall  solves  two  problems  with  the  original  prototype  design. 

•  Eirst,  it  integrates  the  game  story,  physics  gameplay,  and  SEE  levels. 

•  Second,  it  encourages  players  to  finish  levels  in  as  few  tries  as  possible.  It  achieves  these 
ends  by  making  the  "treasure  chest"  the  unit  of  success  throughout  the  game. 

RoboBall  now  begins  with  a  "slide  show"  that  sets  up  the  game  play  conceits.  Players  watch  as  an 
asteroid  swarm  damages  the  Queen's  cargo  ship,  scattering  her  treasure  chests  throughout  the  solar 
system.  This  introductory  movie  to  World  One  furthers  the  story  by  showing  the  game's  crew  on  a 
mission  to  retrieve  some  chests  from  a  nebula  located  in  deep  space. 

This  story  matches  the  game  play  goals  in  both  the  physics  and  SEE  levels.  In  physics  levels,  players 
must  recover  chests  using  the  game’s  RoboBalls.  In  the  SEE  levels,  players  open  the  chests 
recovered  in  the  previous  unit. 

RoboBall  leverages  the  collectible  nature  of  the  chests  to  discourage  "trial  and  error"  gameplay. 

•  At  the  end  of  each  level,  the  game  displays  the  number  of  chests  so  far  collected  in  the  unit. 

•  If  players  fail  three  times  in  a  given  unit,  the  game  resets  progress  to  the  start  of  the  unit,  and 
players  lose  accumulated  chests. 

•  Once  the  player  successfully  completes  a  unit,  the  game  "unlocks"  the  chests,  rewarding  the 
player  with  new  items  on  the  "collectibles"  screen. 

5.15.  ADAPTIVE  GAMEPLAY  ELOW  AND  ADAPTIVE  LEVEL  ELOW 
|5.15.1.  ADAPTIVE  LEVEL  ELOW  ALGORITHM 

The  following  diagrams  summarize  how  Intific’s  Adaptive  Level  Flow  Algorithm  and  Dynamic 
Level  Loading  Algorithm  operate. 
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Adaptive  Level  Flow  Algorithm  Dynamic  Level  Loading  Algorithm 


5.15.2.  DYNAMIC  LEVEL  LOAD  SYSTEM 

This  section  discusses  the  OurSpace  engine's  Dynamic  Level  Loader,  its  use  cases,  and  its 
architectural  requirements,  evolved  during  Phase  II  engineering  efforts. 


I  2. 1.1.1.  OVERVIEW 

OurSpace  has  a  dynamic  level  loader  that  allows  learning  games  to  load  different  types  of 
content  based  upon  the  current  game  state. 

"Game  state"  refers  to  a  combination  of  variables  that  may  come  from  the  portal  from  which 
the  game  was  launched,  the  game  itself,  or  the  current  user's  profile. 

Example:  Student  153  launches  the  latest  version  of  RoboBall.  Student  153  intends  to 
continue  her  game  from  its  last  save  point.  She  has  been  struggling  with  the  game 
recently,  and  the  adaptive  content  algorithm  has  determined  she  needs  extra  help  with  the 
current  learning  target.  When  the  game  starts,  the  dynamic  level  loader  checks  her 
current  learning  goals  in  order  to  choose  the  next  appropriate  level. 
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USE  CASES 

At  a  minimum,  the  dynamic  level  loader  was  required  to  support  the  following  modes  of 
operation: 

1.  Serve  up  a  set  of  demo  levels 

2.  Serve  up  levels  in  a  dynamic  order  based  on  game  state  (e.g.,  return  to  the  last 
checkpoint  after  the  player  dies  3  times) 

3.  Serve  up  levels  in  a  dynamic  order  based  on  player  state  (e.g.,  skipping  content 
related  to  learning  goals  the  student  has  already  mastered) 


I  2. 1 . 1 .2.  ROBOBALL-SPECIFIC  REQUIREMENTS 

In  the  Phase  II  design,  RoboBall  also  required  the  following  features  to  support  its  dynamic 
level  loading  algorithm: 

•  Ability  to  store  learning  goals  associated  with  a  level  in  the  level's  metadata 

•  Ability  to  identify  a  level's  "Unit  Identifier"  in  the  level's  metadata 

•  Ability  to  store  the  level's  type  (Demo,  Assessment,  Exercise,  or  Adaptive)  in  the 
level  metadata 

•  Ability  to  load  all  the  metadata  associated  with  all  the  levels  in  a  given  level  list  in  a 
single  call 


2. 1. 1. 3.  REVISED  LEARNING  TARGETS 

To  support  this  new  Adaptive  design,  we  have  also  broken  down  the  original  learning  targets 
into  smaller  sub-goals.  The  listing  below  shows  all  goals  relating  to  the  content  in  World  0: 

LEARNING  GOALS 

Target  M2:  Students  will  be  able  to  distinguish  the  concepts  of:  position  and 
distance,  time  and  duration,  speed. 

•  M2-I :  The  larger  the  push  or  pull  (impulse)  acting  on  a  body,  the  larger  the 
change  in  the  object's  momentum 

•  M2-2:  An  object  under  constant  acceleration  travels  faster  per  unit  time  in 
each  subsequent  time  interval 

Target  FI:  Students  can  demonstrate  that  a  force  is  a  push  or  pull  that  can  change  an 
object’s  position  and  put  it  into  motion.  BACKGROUND:  The  position  and  motion 
of  an  object  can  be  changed  by  pushing  or  pulling.  An  object  at  rest  will  stay  at  rest 
unless  a  force  is  applied. 

•  EI-I:  a  stationary  mass,  when  pushed  or  pulled,  moves  in  the  direction  of  the 
push  or  pull 

•  El -2:  CLARIFICATION  -  in  the  absence  of  external  forces  like  gravity, 
pushes  and  pulls  behave  identically  in  all  direction 
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Target  F2:  Students  will  demonstrate  understanding  that  the  more  massive  an  object, 
the  more  force  is  needed  to  set  the  object  into  motion  in  the  direction  of  the  force. 
Mass  is  the  amount  of  matter  in  an  object.  Mass  is  not  the  same  as  size.  Force  is  a 
push  or  pull 

•  F2-1:  given  the  same  impulse,  the  heavier  of  two  masses  will  experience  a 
smaller  change  in  motion 

•  F2-2:  given  two  objects  of  equal  mass,  that  which  experiences  the  larger 
impulse  will  experience  a  greater  change  in  motion 

Target  F3:  Students  are  able  to  explain  that  forces  are  composed  of  magnitude  and 
direction.  BACKGROUND:  Magnitude  of  force  is  its  size  and  strength.  Force  always 
has  a  direction. 

•  F3-1  (goal  17):  given  two  impulses  at  right  angles  to  one  another,  students 
can  predict  the  resultant  direction 

•  F3-2  (goal  18):  given  two  opposing  impulses  acting  on  a  single  mass, 
students  correctly  predict  whether  or  not  impulses  cancel 

•  F3-3  (goal  19):  given  unequal  masses  traveling  with  equal  speed,  students 
understand  that  stopping  the  greater  mass  requires  greater  impulse 

•  F3-4  (goal  20):  given  equal  masses  traveling  at  unequal  speeds,  students 
understand  that  stopping  the  faster  mass  requires  greater  impulse 

•  F3-5:  in  the  absence  of  external  forces,  impulses  need  not  be  applied  at  the 
same  time  to  produce  the  same  resultant  momentum 

Target  Gl:  Gravity  in  one  dimension 

•  Gl-1:  an  object  under  the  influence  of  gravity  accelerates  straight  down 

•  Gl-2:  gravitational  acceleration  is  independent  of  mass 

Target  G2:  Gravity  in  two  dimensions 

•  G2-1:  gravity  affects  only  the  vertical  component  of  motion 

•  G2-2:  the  normal  force  offsets  gravity:  the  steeper  the  incline,  the  smaller  the 
offsetting  (normal) 

•  G2-3:  constant  vertical  acceleration,  combined  with  constant  horizontal 
speed,  results  in  parabolic  trajectory  force 

Target  Frl:  Students  can  identify  and  explain  that  friction  manifests  itself  as  a  force, 
and  that  this  force  always  acts  in  a  direction  opposite  to  that  of  a  moving  object. 
BACKGROUND:  An  object  in  motion  will  stay  in  motion  unless  acted  upon  by  an 
outside  source. 

•  Frl-1:  friction  is  a  force  created  by  the  interaction  of  two  surfaces 

•  Frl -2:  frictional  force  usually  opposes  the  motion  of  bodies  involved 

•  Frl -3:  frictional  force  varies  directly  with  mass,  resulting  in  identical 
acceleration  for  objects  of  different  weight 

Target  Fr2:  Students  can  determine,  in  relative  terms,  how  much  friction  a  given 
surface’s  texture  will  pose  to  objects  moving  along  it  (a  lot,  a  little,  somewhere  in 
between 
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•  Fr2-1 :  the  magnitude  of  friction  force  depends  on  the  surfaces  involved  in  the 

interaction 


I  2. 1 . 1 .4.  WORLD  0  LEVEL  PROGRESSION 

This  section  presents  the  learning  progression  for  World  0  (Eorce  and  Motion  in  the  Absence 
of  Gravity  and  Friction).  The  progression  builds  on  itself,  introducing  a  single  concept  with 
each  suite.  In  the  diagrams  below,  each  level  type  is  color-coded  as  follows: 

LEVEL  TYPES  KEY 

•  Demo  Levels  (green):  these  demonstrate  new  game  mechanics.  Students 
cannot  fail  these.  The  game  disregards  data  from  these  levels. 

•  Assessment  (Superstar)  Levels  (pink):  these  present  challenging  problems 
using  only  previously-demonstrated  mechanics.  Students  are  allowed  only  one 
attempt  to  solve  the  level. 

•  Exercise  Levels  (tan):  these  present  "normal"  problems  using  only 
previously-demonstrated  mechanics.  These  levels  provide  the  bulk  of  student 
performance  data. 

•  Adaptive  levels  (cyan):  these  are  additional  exercise  levels  that  appear  only  if 
the  player  has  performed  below  80%  on  associated  learning  target 


Game  Progression 

IMPULSE  WORLD,  UNIT  I 

(IMAGES  BEGIN  ON  THE  EOLLOWING  PAGE) 


Intific  Inc. 


30 

Data  contained  on  this  page  is  subject  to  restrictions  on  cover  and  notice  page. 


Alternate  Reality  Teaching:  OurSpace 


N00014-11-C-0593 


world:  0  Physical  Rules:  no  gravity,  no  friction 

Unit:  1  Learning  Goals:  Fl-1,  Fl-2 
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world:  0  Physical  Rules:  no  gravity,  no  friction 

^^Unlt:  1  Learning  Goals:  Fl-1,  Fl-2 
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IMPULSE  WORLD,  UNIT  2 
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world:  0  Physical  Rules:  no  gravity,  no  friction 

^Unit:  2_ Learning  Goals:  Fl-1,  Fl-2,  F2-2,  M2-1 
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IMPULSE  WORLD,  UNIT  3 
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For  completeness, 
we  should  rotate 
this  level  90 
degrees  when  making 
making  it  in  the  editor. 
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IMPULSE  WORLD,  UNIT  4 


World:  0  Physical  Rules:  No  gravity,  no  friction 

Unit:  4  Learning  Goals:  Fl-1,  Fl-2,  F3-1 


Exercise  1 


Exer^  2 


Exerd^S 


world:  0  Physical  Rules:  no  gravity,  no  friction 

Unit:  4  Learning  Goals:  Fl-1,  Fl-2,  F3-1 
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IMPULSE  WORLD,  UNIT  5 


|>r1d:  0  Physical  Rules:  No  gravity,  no  friction 

|nit:  5  Learning  Goals:  Fl-1,  Fl-2,  F2-1,  F2-2,  F3-1,  F3-5 


World:  0  Physical  Rules:  No  gravity,  no  friction 

Unit:  5  Learning  Goals:  Fl-1,  Fl-2,  F2-1,  F3-1,  F3-5 
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IMPULSE  WORLD,  UNIT  6 


world : 

0 

Physical 

Rules : 

NO  gravity, 

no  friction  ^ 

Unit: 

_ 

6 

Learning 

Goals : 

Fl-1,  Fl-2, 

F2-1,  F2-2,  F3-2,  M2-1j 
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IMPULSE  WORLD,  UNIT  7 


world : 
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Physical 

Rules : 

NO  gravity, 
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Unit: 
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Fl-1,  Fl-2, 

F2-1,  F2-2,  F3-1,  F3-2 
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2. 1.1. 5.  ASSESSMENT  ALGORITHM  EOR  ADAPTATION 

The  following  section  further  describes  the  algorithm  used  to  determine  which  Adaptive 
levels  to  assign  to  RoboBall  players. 

ASSESSMENT,  ADAPTATION,  AND  LIVES 

RoboBall  provides  two  systems  for  managing  "Adaptive  Content"  —  Lives  and  Dynamic 
Level  Loading: 

•  The  Lives  system  awards  the  player  three  "lives"  at  the  start  of  each  Unit 

o  Lailure  of  a  level  results  in  the  loss  of  one  life 

o  Loss  of  all  lives  results  in  automatic  reset  to  the  first  level  in  the  Unit  and  the 
restoration  of  the  three  lives 

o  This  system  ensures  a  minimum  mastery  of  concepts  before  allowing  players 
to  proceed  to  later  Units,  which  addresses  the  game's  current  "scaffolding" 
problem 

•  Dynamic  Level  Loading  examines  the  player's  performance  at  the  end  of  each  Unit's 
mandatory  content 

o  If  the  player  needs  additional  help,  the  game  inserts  optional  levels  into  the 
level  flow 

o  If  the  player  needs  no  additional  help,  the  game  proceeds  to  the  Unit  Success 
screen  and  awards  bonus  Chests  for  unplayed  content 

ADAPTATION  ALGORITHM 

RoboBall  uses  the  following  algorithm  to  determine  if  players  need  additional  help  at  the 
end  of  a  Unit's  mandatory  content: 

•  At  the  end  of  a  level,  the  game  records  an  "attempt"  for  each  learning  target 
associated  with  that  level 

•  The  game  tracks  only  the  last  5  attempts  per  learning  goal,  represented  by  a 
circular  buffer 

•  If  the  player  successfully  completed  the  level,  the  game  records  the  success  in 
another  circular  buffer 

•  At  the  end  of  the  Unit's  mandatory  content,  the  game  computes  the  player's  "pass 
rating"  (e.g.,  75%)  for  all  learning  targets 

•  Per  goal,  if  the  "pass  rating"  falls  below  a  specified  threshold,  the  game  flags  that 
goal  as  a  'review  subject' 

•  The  game  then  assigns  optional  levels  whose  learning  goals  match  any  of  the 
'review  subjects' 
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LEVEL 


SCREENSHOT 


Unit  1 


Objective:  Adding  a  force  (a  push)  so  the  ball  will  hit  the  treasure  chest 


NOTES 


In  these  introductory  levels,  the  force  magnitude  and  mass  of  the  ball  are  fixed  (i.e.,  ‘big  force’  and  small  mass  ball) 


However,  the  direction  of  the  force  (e.g.,  horizontal  or  vertical)  does  vary,  and  this  unit  focuses  on  the  direction  of  the  force. 


Students  learn  that  the  force  can  change  the  ball’s  position  and  put  it  in  motion. 


1-1, 

1-2 


1. 

2. 


3. 

4. 


In  Level  1 ,  the  game  models  what  to  do  by  showing 
the  punching  glove  move  to  the  appropriate  location 

Students  can  only  place  objects  in  the  toolbox  at 
locations  marked  by  this  icon: 


Instruct  students  to  push  the  “play”  button  once  they 
have  place  the  object(s)  in  the  desired  locations. 
There  is  only  one  possible  answer  option  for  the  first 
two  levels  to  get  students  acquainted  to  the  game 
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^0 


1-3 


Ask  the  student  where  they  should  place  the 
treasure  chest  so  the  ball  will  hit  it.  If  needed, 
remind  the  student  that  they  can  only  place  the 
chest  on  one  of  the  blue-purple  circles. 

There  are  multiple  potential  locations  for  where  to 
place  the  treasure  chest.  Student  needs  to 
understand  the  direction  of  the  force  (vector). 
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1 .  Student  must  decide  on  the  direction  of  the  force 
needed  to  make  the  ball  hit  the  treasure  chest.  Ask 
the  student  if  they  need  to  make  the  ball  move 
upward  or  downward  (or  right  vs.  left),  and  which 
punching  glove  they  should  pick  to  achieve  this. 
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Screenshot 


Notes 


Level 


Unit  2  Objective:  Unit  2  introduces  the  student  to  the  concept  of  magnitude,  which  is  the  size  and  strength  of  the  force.  There  are  two  force  magnitudes  in 
this  game  (i.e.,  big  and  small). 

Students  will  be  able  to  distinguish  the  concepts  of  position  and  distance,  and  understand  that  there  is  a  time  component  when  moving 
from  one  position  to  another. 


^  ♦*  ; 


/' 


1 .  Level  introduces  player  to  the  concept  of  magnitude,  which  is  the  size 
and  strength  of  the  force. 

2.  Students  observe  and  compare  what  happens  when  the  ball  is  hit  by  a 
punching  glove  with  a  small  force  (i.e.,  shorter  arrow)  versus  a 
punching  glove  with  a  big  force  (i.e.,  longer  arrow) 

3.  Can  prompt  student  by  asking  them  which  ball  reached  the  treasure 
chest  first  and  why  they  think  this  is  the  case 

a.  Student  should  understand  that  a  bigger  force  gets  the  ball  to 
the  treasure  chest  “faster”,  and  therefore  reaches  the  treasure 
chest  in  a  shorter  amount  of  time  (in  comparison  to  the  ball  hit 
with  a  small  force). 
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1. 

2. 

3. 


1. 


In  Unit  1 ,  students  learned  about  direction,  and  this  concept  reappears 
in  Unit  2.  The  purple  arrows  indicate  that  once  the  ‘play’  button  is 
pushed,  the  objects  will  move  in  the  direction  of  the  arrows. 

The  ball  cannot  hit  the  moving  comet,  but  must  hit  the  moving  treasure 
chest. 

There  is  only  one  possible  (and  correct)  option  in  this  level.  The  student 
should  observe  and  remember  what  happens  when  the  ball  is  hit  with  a 
big  force. 


There  is  only  one  possible  (and  correct)  option  in  this  level.  The  student 
should  observe  and  remember  what  happens  when  the  ball  is  pushed 
with  a  ‘small  force’ 
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1. 


2. 


1. 

2. 


Drawing  on  observations  and  knowledge  gained  from  the  previous 
levels,  the  student  needs  to  place  the  treasure  chest  on  the  appropriate 
location  (i.e.,  marker  in  first  column)  so  that  the  ball  hit  with  a  small 
force  will  hit  the  treasure  chest. 

If  student  gets  stumped,  instruct  them  to  think  about  what  happened  in 
the  previous  levels  they  just  completed  (comparing  how  far  and  how 
fast  the  ball  will  travel  when  hit  with  a  small  or  big  force).  And,  remind 
student  that  the  location  targets  will  move  in  the  direction  of  the  purple 
arrows  once  they  hit  play. 


Student  needs  to  select  whether  a  small  or  big  force  is  needed  to  get 
the  ball  to  miss  the  comet,  but  hit  the  treasure  chest. 

Instruct  student  to  use  information  from  the  previous  levels  to  decide 
which  sized  force  magnitude  is  needed.  Based  on  the  previous  level  (2- 
4),  a  student  should  know  that  a  small  force  would  lead  the  ball  to  hit 
the  comet  (as  the  placement  is  identical  to  the  previous  level). 
Therefore,  the  big  force  is  the  correct  answer. 
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2-8 

‘  0  ir  0% 

Similar  to  levels  2-5  and  2-6,  except  this  time,  instead  of  selecting  the 
correct  force  magnitude,  the  student  needs  to  decide  where  to  place 
the  ball. 
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Level 


Screenshot 


Notes 


Unit  3  Objective: 

Unit  3  introduces  the  concept  of  mass  by  varying  the  mass  of  the  balls  (i.e.,  small  and  large  mass  balls) 

Students  are  also  introduced  to  the  concept  of  force  as  a  vector  (i.e.,  forces  are  composed  of  magnitude  &  direction). 


Students  will  be  able  to  describe  the  motion  of  the  balls  addressing  components  of  position,  distance,  and  time  and  using  words  such  as  ‘faster,’ 
‘slower,’  ‘more,’  and  ‘less.’ 


Students  will  demonstrate  understanding  that  for  an  object  to  accelerate  at  a  given  rate;  more  force  is  needed  for  objects  with  greater  mass  than  for 
objects  with  lesser  mass. 

Student  will  understand  that  when  force  magnitude  is  held  constant,  the  ball  with  less  mass  will  travel  faster  than  the  ball  with  greater  mass. 


3-1 


1. 

2. 


Student  is  able  to  observe  and  compare  what  happens  when  a  big 
force  hits  a  ball  with  greater  mass  versus  a  ball  with  less  mass 

Prompt  students  to  identify  which  ball  reached  the  treasure  chest  first, 
and  to  explain  why  they  think  this  is  the  case 

a.  Student  should  understand  that  the  ball  with  less  mass  travels 
faster  (than  the  ball  of  greater  mass)  when  provided  with  a 
push  of  the  same  force  magnitude 
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1. 

2. 


1. 


Player  is  able  to  observe  and  compare  what  happens  when  a  small 
force  hits  a  ball  with  greater  mass  versus  a  ball  with  less  mass 

Prompt  students  to  identify  which  ball  reached  the  treasure  chest  first, 
and  to  explain  why  they  think  this  is  the  case 

a.  Student  should  understand  that  the  ball  with  less  mass  travels 
faster  (than  the  ball  of  greater  mass)  when  provided  with  a 
push  of  the  same  force  magnitude 


After  placing  the  punching  gloves  in  the  designated  locations,  prompt 
students  to  describe  and  explain  what  happened,  and  why  they  think 
this  is  the  case 

a.  By  placing  both  ‘big  force’  punching  gloves  (vectors)  in  the 
designated  locations,  the  student  will  observe  that  when 
holding  force  magnitude  constant,  the  ball  with  greater  mass 
will  travel  slower  (hitting  the  treasure  chest  that  is  closer),  and 
how  the  ball  with  less  mass  will  travel  faster  (hitting  the 
treasure  chest  that  is  farther  away). 
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Levels  3-4  and  3-5  are  similar  to  levels  3-6  through  3-8 


3-8 


Levels  3-4  and  3-5  are  similar  to  levels  3-6  through  3-8 
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Level 


Screenshot 


Notes 


Unit  4  Objective: 

Students  will  learn  that  resultant  forces  arise  when  two  or  more  forces  In  different  directions  are  added  together,  and  that  when  applied  to  an  object, 
the  object’s  resulting  direction  of  motion  and  rate  of  acceleration  will  be  determined  by  the  magnitude  and  direction  of  motion  of  the  resultant  (net) 
force  applied. 


Student  should  also  learn  that  when: 


•  Horizontal  force  >  Vertical  force  ->  Object’s  path  is  closer  to  X-axis 


•  Horizontal  force  =  Vertical  force  ->  Object’s  path  follows  a  45°  angle 

•  Horizontal  force  <  Vertical  force  ->  Object’s  path  Is  closer  to  Y-axis 


Unit  4  game  play  should  dispel  the  misconception  that  the  motion  of  an  object  is  always  determined  by  the  last  force  applied  to  it. 
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•  Student  must  decide  where  to  place  the  treasure  chest  based  on  the 
projected  path  of  the  ball  given  the  resultant  net  force  applied 

o 

•  Student  needs  to  understand: 

Horizontal  force  <  Vertical  force  ->  Object’s  path  is  closer  to  Y-axis 

1 
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Q:  What  will  happen  if  we  just  give  the  ball  a 
big  force  this  way  (going  down)? 

A:  Ball  will  move  straight  down. 


Now,  this  glove  is  also  going 
to  give  the  ball  a  small  force 
this  way  (going  left). 

Let's  say,  since  It  is  a  small 
force,  it  Is  going  to  push  It 
over  one  square. 


Small  force  i 
=  1  square  \ 


And  since  this  glove 
gives  a  big  forces  it  will 
push  the  ball  over  two 
squares. 


Big  force  s 
2  squares 
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So  imagine,  ball  goes  down 
for  two  squares,  and  boom, 
one  small  square  this  way. 
Then  two  big  squares  down, 
and  boom,  another  one  small 
square  this  way. 

Which  target  will  the  ball  hit? 


Small 

i  Big 

Small  fr— I-  ^ 
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•  Student  must  decide  which  vectors  are  needed  (and  where  they  should 

be  placed)  in  order  to  ensure  that  the  ball  hits  the  treasure  chest. 

•  Student  needs  to  understand: 

Horizontal  force  >  Vertical  force  ->  Object’s  path  is  closer  to  X-axis 
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The  only  glove  that  points  to  the  left  will  give  a  small  force.  Again,  let's  say  that  since  it  Is  a  small 
force,  it  is  going  to  push  it  over  one  square. 


Let's  see  what  happens  if 
we  use  a  small  force  In 
the  up  direction. 
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Let's  see  what  happens  if  we 
use  a  big  force  in  the  up 
direction. 
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•  Student  must  decide  which  vectors  are  needed  (and  where  they  should 

I  C^i 

be  placed)  in  order  to  ensure  that  the  ball  hits  the  treasure  chest. 

•  Student  needs  to  understand: 

< 

Horizontal  force  =  Vertical  force  ->  Object’s  path  follows  a  45°  angle 

-t  :  ^  O 

1 
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•  Student  must  decide  where  to  place  the  treasure  chest  based  on  the 

projected  path  of  the  ball  given  the  resultant  net  force  applied 

•  Student  needs  to  understand: 

Horizontal  force  <  Vertical  force  ->  Object’s  path  is  closer  to  Y-axis 

& 

/^i 
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Student  must  decide  where  to  place  the  ball  to  ensure  that  the  ball  hits  the 
treasure  chest.  Needs  to  understand  which  net  force  will  place  the  ball 
on  the  correct  trajectory 

Student  needs  to  understand: 

Horizontal  force  =  Vertical  force  ^  Object’s  path  follows  a  45°  angle 
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•  Student  must  decide  where  to  place  the  ball  to  ensure  that  the  ball  hits  the 

treasure  chest.  Needs  to  understand  which  net  force  will  place  the  ball 
on  the  correct  trajectory 

•  Student  needs  to  understand: 

Horizontal  force  >  Vertical  force  ^  Object’s  path  is  closer  to  X-axis 
(answer  is  top  location;  the  bottom  location  would  require  =  forces  to 
create  a  45-degree  pathway  to  hit  the  treasure  chest) 


67 


Data  contained  on  this  page  is  subject  to  restrictions  on  cover  and  notice  page. 


Intific  Inc. 


Alternate  Reality  Teaching:  OurSpace 


N00014-11-C-0593 


4-8 


Student  must  decide  which  vectors  and  net  force  is  needed  to  ensure  that 
the  ball  hits  the  treasure  chest. 

Student  needs  to  understand: 

Horizontal  force  >  Vertical  force  ->  Object’s  path  is  closer  to  X-axis 


Level 


Screenshot 


Notes 


Unit  5 


Objective: 

In  Unit  4,  two  forces  were  provided  simultaneously.  In  contrast,  in  Unit  5,  forces  are  provided  sequentially  (one  after  the  other)  in  most  cases. 
Similar  rules  still  apply: 

•  Horizontal  force  >  Vertical  force  ->  Object’s  path  is  closer  to  X-axis 

•  Horizontal  force  =  Vertical  force  ->  Object’s  path  follows  a  45°  angle 

•  Horizontal  force  <  Vertical  force  ->  Object’s  path  is  closer  to  Y-axis 
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Game  play  should  dispel  the  misconception  that  the  motion  of  an  object  is  always  determined  by  the  last  force  applied  to  it. 
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•  Student  must  decide  where  to  place  the  treasure  chest  based  on  the 

projected  path  of  the  ball  given  the  resultant  net  force  applied 

•  Student  needs  to  understand: 

Horizontal  force  <  Vertical  force  ^  Object’s  path  is  closer  to  Y-axis 


5-2 


•  Student  must  decide  which  vectors  and  what  net  force  is  needed  to 

ensure  that  the  ball  hits  the  treasure  chest. 

•  Student  needs  to  understand: 

Horizontal  force  =  Vertical  force  ^  Object’s  path  follows  a  45°  angle 
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Student  must  decide  which  vectors  and  what  net  force  is  needed  to 
ensure  that  the  ball  hits  the  treasure  chest. 

Student  needs  to  understand: 

Horizontal  force  >  Vertical  force  ->  Object’s  path  is  closer  to  X-axis 


Student  must  decide  which  vectors  and  what  net  force  is  needed  to 
ensure  that  the  ball  hits  the  treasure  chest. 

Student  needs  to  understand: 

Horizontal  force  =  Vertical  force  ^  Object’s  path  follows  a  45°  angle 
As  in  Unit  4,  vectors  are  applied  to  ball  simultaneously 
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•  Student  must  decide  which  vectors  and  what  net  force  is  needed  to 

ensure  that  the  ball  hits  the  treasure  chest. 

•  Student  needs  to  understand: 

Horizontal  force  =  Vertical  force  ->  Object’s  path  follows  a  45°  angle 


•  Student  must  decide  which  vectors  and  what  net  force  is  needed  to 

ensure  that  the  ball  hits  the  treasure  chest. 

•  Student  needs  to  understand: 

Horizontal  force  <  Vertical  force  ^  Object’s  path  is  closer  to  Y-axis 
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Student  must  decide  which  vectors  and  what  net  force  is  needed  to 
ensure  that  the  ball  hits  the  treasure  chest. 

Student  needs  to  understand: 

Horizontal  force  >  Vertical  force  ->  Object’s  path  is  closer  to  X-axis 


Student  must  decide  which  vectors  and  what  net  force  is  needed  to 
ensure  that  the  ball  hits  the  treasure  chest. 

Student  needs  to  understand: 

Horizontal  force  <  Vertical  force  ^  Object’s  path  is  closer  to  Y-axis 
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Level 


Screenshot 


Notes 


Unit  6 


Objective: 

Introduces  two  opposing  forces  (forces  in  the  opposite  direction) 

Students  will  be  able  to  describe  the  motion  of  the  balls  addressing  components  of  position,  distance,  and  time  and  using  words  such  as  ‘faster,’ 
‘slower,’  ‘more,’  and  ‘less.’ 

Student  needs  to  think  about  the  timing  and  how  fast  the  ball  will  move  when  hit  with  differing  vectors. 

Game  play  should  dispel  the  misconception  that  the  motion  of  an  object  is  always  determined  by  the  last  force  applied  to  it. 


60 

O'  V' 

Q 

•  In  levels  6-1  to  6-7,  the  student  must  decide  where  to  place  the  treasure 

chest  in  order  to  ensure  that  the  ball  hits  the  treasure  chest.  Student 

needs  to  think  about  the  timing  and  how  fast  the  ball  (of  small  or  large 

10 

mass)  will  move  when  hit  with  differing  vectors. 

n 

fi 

/ 

n 

1 

74 


Data  contained  on  this  page  is  subject  to  restrictions  on  cover  and  notice  page. 


Intific  Inc. 


Alternate  Reality  Teaching:  OurSpace 


N00014-11-C-0593 


J  » 

€0 

t 

1 

«  .A  ^ 

|H 

■  .* 

? 

1 

75 


Data  contained  on  this  page  is  subject  to  restrictions  on  cover  and  notice  page. 


Intific  Inc. 


Alternate  Reality  Teaching:  OurSpace 


N00014-11-C-0593 


6-4 


@©  r\  ,  •> 

y  « 

/  to 


@©  <'^1 
\ 

♦ 

.>s 

V 

mi 

«» 

• 

6-5 


76 


Data  contained  on  this  page  is  subject  to  restrictions  on  cover  and  notice  page. 


Intific  Inc. 


Alternate  Reality  Teaching:  OurSpace 


N00014-11-C-0593 


6-6 


@@ 

©  ''i* 

I  * 

% 

) 

1 

#  o 

fl 

A 

6-7 


©© 

"1* 

m 

J 

i . 

/ 

t 

t 

77 


Data  contained  on  this  page  is  subject  to  restrictions  on  cover  and  notice  page. 


Intific  Inc. 


Alternate  Reality  Teaching:  OurSpace 


N00014-11-C-0593 


Level 


Screenshot 


Notes 


Unit 

7 


Objective: 

Introduces  a  sequence  force  combinations  (with  some  involving  simultaneous  forces) 
opposing  forces  (forces  in  the  opposite  direction) 


Students  will  be  able  to  describe  the  motion  of  the  ball  as  equal  forces  cancel  each  other  out  and  prevailing  forces  push  the  ball  along. 

Student  needs  to  think  about  how  fast  the  ball  will  move  when  hit  with  differing  vectors,  and  determine  if  the  next  force  that  acts  upon  the  ball  will 
redirect  it  or  stop  it. 


Game  play  should  dispel  the  misconception  that  the  motion  of  an  object  is  always  determined  by  the  last  force  applied  to  it. 
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5.15.3.  GAME  REVISION  HISTORY 


Intific's  ENGAGE  Game  Revision  Histor 

y 

Deliverable 

Defining  Features 

Physics 

Levels 

SEL 

Levels 

CRESST  Tested 

Notes 

Closed  Alpha 
-March  2013 

Updated  External  Portal  -  OurSpace 

Converted  Flash  prototype  levels  to  html5 

35 

1 

YES  -  80  students 

The  updated  OurSpace  portal  enabled 
CRESST  and  our  developers  to  use  the 
same  site  for  testing,  data  collection  and 
level  editing. 

Closed  Beta 
-May  2013 

Converted  Art  to  modular  pieces 

Improved  Log  files 

Improved  Frame  Rate 

Improved  Editor  Functionality 

83 

6 

NO  -  CRESST  did 

not  test  this  version 
of  the  game. 

Making  the  art  pieces  modular  enabled 
our  developers  to  increase  their  level 
editing  producti\ity.  It  also  allow’ed  us  to 
iterate  on  our  designs  more  rapidly. 

Open  Beta 
-  Sept  2013 

AdaptiV'C  Learning 

Integrated  Pre  and  Post  Test  Assessments 
Improved  Waming/Error  Messaging 

Award  System 

Improved  Physics  Determinism 

Added  Storyboard  Intro  Movies 

52 

6 

YES  -  41  students 

The  adaptiv'e  learning  design  and  the 
integrated  pre  and  post  tests  are  the 
hallmark  of  w'hat  the  research  version  of 
our  game  is  based  on.  With  these  features 
external  research  agencies  can  easily 
collect  efficacy  and  assessment  data. 

Final  Build 

-  Feb  2014 

Improved  Waming/Error  Messaging 

Scaled  Serv^er  Solution  (Amazon  Serv^er) 
Increased  Frames  Per  Second  fi-om  30  to  60 

52 

6 

YES  -  20  students 

Scaling  our  serv’er  solution  allow’ed  us  to 
support  a  larger  user  base.  This 
improvement  enabled  us  to  support 
thousands  of  users  instead  of  hundreds. 

SpaceForce 
-March  2014 

Public  Facing  Version  of  the  Game 

Partnered  with  BrainPOP 

Re\ised  User  Interface  for  BrainPOP's  audience 

52 

NO  -  Available  to 
the  public. 

Our  partnership  with  BrainPOP  has 
allow' ed  us  to  transition  our  game  fi-om  a 
research  project  to  a  consumer  product. 
Thousands  of  students  play  it  every  day. 

Figure  1.0  -  OurSpace  game  revision  history  diagram 
5.15.4.  CONEERENCES 

During  Phase  2  the  Intific  team  attended  several  education  conferences.  The  reason  for  this  was 
two-fold;  social  networking  and  showcasing  our  first  OurSpace  product.  Below  is  a  list  of 
conferences  with  highlights  from  each  event. 

DARPA  120  Demo  Day  at  the  Pentagon 

The  Demo  Day  at  the  Pentagon  was  an  opportunity  for  DARPA  to  showcase  its  Innovation  and 
Information  Offices  work.  There  were  29  Programs  represented  at  the  demo  day  with  well  over 
100  exhibits.  There  were  one  to  two  thousand  uniformed  and  civilian  attendees  from  the  Pentagon 
who  walked  around  the  courtyard  to  look  at  each  DARPA  project.  Intific  had  at  least  100  people 
stop  by  to  discuss  the  game  and  find  out  what  it  was  about  and  where  they  could  play  it.  We 
passed  out  information  and  gave  many  demos.  A  member  of  the  press  from  a  tech  magazine 
interviewed  Col.  Ragsdale  and  a  member  of  the  Intific  team.  He  also  played  the  game  for  several 
minutes.  He  was  taking  notes  and  took  some  contact  information  and  a  handout.  There  may  be  an 
article  released  about  our  game.  Many  parents  and  former  teachers  we  very  excited  about  what 
we're  doing.  Even  military  personnel  liked  our  approach  to  getting  kids  excited  about  learning;  by 
developing  compelling  and  highly  interactive  educational  games.  The  event  was  a  great  success 
for  our  team. 
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During  the  event  we  also  met  a  few  partners  that  we  work  with  on  other  DARPA  programs  such 
as  ABM  on  the  DCAPS  Program.  We  also  got  a  chance  to  speak  with  other  ENGAGE  program 
performers;  Carnegie  Mellon  University,  University  of  Washington  and  our  TA-2  partner; 
CRESST.  The  event  was  a  great  opportunity  to  meet  people  working  on  similar  products  in  the 
same  field. 

CRESST  Center  for  Advanced  Technology  in  Schools  conference  (CATS) 

On  April  29-30,  2014,  Dr.  Amy  Kruse  attended  the  CRESST  Center  for  Advanced  Technology  in 
Schools  conference.  Dr.  Kruse  represented  the  Intific  education  efforts  interacting  with  the 
hundreds  of  participants,  including  our  CRESST  colleagues.  Dr.  Kruse  presented  a  Teddy  Talk 
on  the  future  of  education  games,  particularly  those  influenced  and  inspired  by  neuroscience 
research.  Dr.  Kruse  also  demonstrated  the  Intific  SpaceEorce  game  to  the  conference  attendees, 
directing  them  both  to  the  new  KADDEO  education  page  and  the  BrainPOP  Jr.  sites.  The 
response  was  enthusiastic  and  many  educators  in  attendance  were  excited  about  both  the  existing 
games  and  the  architecture  built  by  the  DARPA  ENGAGE  program.  Eollow  up  from  the  meeting 
will  include  new  funding  and  teaming  opportunities  within  the  educational  research  and 
development  area. 

STEM  Symposium 

We  had  the  opportunity  to  participate  in  a  STEM  Symposium  in  Northern  VA  in  April  2014. 
Thousands  of  kids  (and  some  type-A  parents)  showed  up  to  talk  about  education  and  interact  with 
our  games. 

We  wish  we  had  video  -  watching  students  PLAY  the  game  and  react  to  it. . .  truly  awesome.  I 
couldn’t  have  been  more  proud!  Most  of  the  day  there  was  a  LINE  to  play  . . .  The  game  was 
EQUALLY  popular  with  boys  and  girls.  And  some  really  tiny  kids  made  great  progress.  It  was 
also  neat  to  see  the  adaptive  aspect  in  play  -  you  could  tell  when  a  student  hadn’t  gotten  the 
concept! 

There  were  actually  some  parents  that  had  to  DRAG  their  kids  away  from  playing  it  (‘‘Just  one 
more  sector,  mom,  I’m  almost  done. . .”) 

South  by  South  West  Education  conference  (SXSWedu) 

On  March  4*  and  5*  we  joined  BrainPOP  at  the  South  by  South  West  Education  conference. 
During  our  presentations  we  discussed  the  game  development  process  used  at  Intific  and  spoke 
about  the  research  and  planning  that  goes  into  developing  an  educational  game  for  DARPA.  At 
the  conference  we  demoed  our  game  and  handed  out  updated  one-sheets  and  activity  books.  We 
received  a  lot  of  compliments  on  how  professional  our  game  looks.  Many  people  wanted  to  know 
what  technology  we  used  to  build  the  game.  They  were  impressed  when  we  told  them  that  we 
developed  the  engine  ourselves. 

Texas  Computer  Education  Association  conference  (TCEA) 

On  Eebruary  5*  and  6*  Intific  attended  the  Texas  Computer  Education  Association  conference 
(TCEA)  with  BrainPOP.  At  the  conference  we  showcased  our  game  and  discussed  the 
educational  game  development  process  that  we  use  at  Intific.  We  handed  out  one-sheets 
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containing  information  about  the  ENGAGE  program  and  our  educational  game.  We  received 
many  positive  comments  about  the  professional  quality  of  the  game.  Our  development  team  is 
proud  of  how  far  this  product  has  come  over  the  past  couple  of  years. 

5.16.  ROBOBALL  DATA  COLLECTION 

This  section  summarizes  the  changes  in  RoboBall  that  are  relevant  to  data  collection  and  analysis 
efforts.  Included  are  breakdowns  of  the  features  added  to  the  final  software  deliverable  to  facilitate 
efficacy  and  assessment  tests.  They  are  as  follows:  new  game  structure,  new  testing  methods,  new 
player  incentives.  Please  see  the  RoboBall  Open  Beta  Documentation  FINAL  for  more  detail 
(included  in  Final  Documentation  Deliverable  Package). 

|5.16.1.  NEW  GAME  STRUCTURE 

RoboBall  Open  Beta  is  divided  into  "Units”  that  cover  specific  learning  targets,  and  the  units  are 
subdivided  into  level  groups  with  different  pedagogical  aims.  The  new  level  types  are: 

1.  Demo  Levels  —  these  introduce  new  game  mechanics  and  have  only  one  possible 
configuration. 

2.  Exercise  Levels  —  these  levels  pose  new  physics  problems  in  their  simplest  form. 

3.  Adaptive  Levels  —  additional  "exercise"  levels  that  frame  previous  problems  in  new  ways. 
Students  will  only  see  adaptive  levels  if  they  have  failed  more  than  20%  of  the  exercise  levels 
for  a  given  set  of  learning  goals. 

4.  Assessment  Levels  —  please  see  "New  Testing  Methods"  below  for  details  about  this  level 
type. 

5.  SEL  Levels  —  the  "social  and  emotional  learning"  levels  from  the  earlier  version  of 
RoboBall. 

Please  see  above  for  a  listing  of  all  units,  the  levels  that  comprise  them,  and  the  learning  targets 
associated  with  each.  Currently,  we're  not  logging  the  level  type,  but  we  could  add  this. 

|5.16.2.  NEW  TESTING  METHODS 

As  mentioned  above,  Intific  has  introduced  "Assessment  Levels"  to  gauge  student  learning  in  a 
crowd-sourced  environment.  Students  play  each  assessment  level  twice:  first,  as  a  pre-test  before 
exposure  to  a  new  concept;  second,  after  completing  all  the  exercise  and  adaptive  levels  relevant 
to  the  new  concept.  We  have  tried  to  place  at  least  seven  levels  between  each  pre-  and  post-test  to 
prevent  students  from  answering  from  memory.  We  have  added  a  field  to  the  logs  to  indicate 
when  the  student  is  playing  an  assessment  level. 

|5.16.3.  NEW  PLAYER  INCENTIVES 

We  also  introduced  two  player  incentives  to  combat  success  through  "trial  and  error."  The  first  is 
"lives."  The  player  starts  each  unit  with  3  crew  members.  Each  time  the  player  fails  a  level, 
another  crew  member  "sits  out."  When  all  members  have  failed,  the  game  restarts  at  the 
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beginning  of  the  unit.  This  mechanic  feeds  into  the  "prizes”  system,  which  rewards  players  for 
success.  Each  time  the  player  completes  a  level,  he  or  she  earns  the  chests  recovered  in  that  level. 
Recovered  chests  remain  locked  until  the  player  completes  the  SEE  level  at  the  end  of  the  unit.  At 
that  point,  the  game  unlocks  the  chests,  adding  new  "stickers"  to  the  collectibles  screen. 


6.  EXTENSION  OF  THE  OURSPACE  GAME  APPROACH 


As  noted  earlier,  OurSpace  is  a  broad  and  ambitious  vision  for  new  types  of  digital  learning.The  data 
collection,  crowdsourced  verification,  and  adaptability  tool  approach  of  these  systems  can  permit  a  broad 
extension  of  this  concept  beyond  the  RoboBall  force,  motion  and  gravity  learning  games.  This  sections 
explores  some  of  the  exciting  opportunities  before  us  in  the  near  future. 

Intific  is  aggressively  exploring  future  opportunities  to  expand  these  technologies  and  welcomes  the 
opportunity  to  also  develop  future  applications  of  these  approaches  in  the  STEM  and  SEE  domains. 

6.1.  HOW  TO  INTEGRATE  ROBOBALL  INTO  THE  CLASSROOM 

The  concepts  developed  in  RoboBall  can  also  be  coupled  with  both  science  and  mathematics 
curriculum.  RoboBall  maps  onto  the  Next  Generation  Science  Standards  (NGSS)  (NSE,  2013).  The 
game  mechanics  in  RoboBall  are  consistent  with  the  Performance  Expectations  contained  in  Table  4, 
covers  the  NGSS  Disciplinary  Core  Ideas  contained  in  Table  5,  and  requires  the  use  of  the  Science 
and  Engineering  practices  contained  in  Table  6. 

Table  1 


NGSS  Performance  Expectations  Related  to  RoboBall 


Performance  Expectations 

K-PS2- 

1 

Plan  and  conduct  an  investigation  to  compare  the  effects  of  different  strengths  or  different 
directions  of  pushes  and  pulls  on  the  motion  of  an  object. 

K-PS2- 

2 

Analyze  data  to  determine  if  a  design  solution  works  as  intended  to  change  the  speed  or 
direction  of  an  object  with  a  push  or  a  pull. 

3-PS2- 

1 

Plan  and  conduct  an  investigation  to  provide  evidence  of  the  effects  of  balanced  and  unbalanced 
forces  on  the  motion  of  an  object. 

MS- 

PS2-2 

Plan  and  conduct  an  investigation  to  provide  evidence  that  the  change  in  an  object’s  motion 
depends  on  the  sum  of  the  forces  on  the  object  and  the  mass  of  the  object. 
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Table  2 


NGSS  Disciplinary  Core  Ideas  covered  in  RoboBall 


Disciplinary  Core  Ideas 

Related  Performance 

Expectations 

Forces  and 

Pushes  and  pulls  can  have  different  strengths  and 

K-PS2-1 

Motion 

directions. 

K-PS2-2 

Pushing  and  pulling  on  an  object  can  change  the  speed  or 

K-PS2-1 

direction  of  its  motion  and  can  start  or  stop  it. 

K-PS2-2 

Each  force  acts  on  one  particular  object  and  has  both 
strength  and  a  direction.  An  object  at  rest  typically  has 
multiple  forces  acting  on  it,  but  they  add  to  give  zero  net 
force  on  the  object.  Forces  that  do  not  sum  to  zero  can 
cause  changes  in  the  object’s  speed  or  direction  of  motion. 

3-PS2-1 

The  motion  of  an  object  is  determined  by  the  sum  of  the 
forces  acting  on  it;  if  the  toil  force  on  the  object  is  not  zero, 
its  motion  will  change. 

MS-PS2-2 

Energy 

The  faster  a  given  object  is  moving,  the  more  energy  it 

K-PS2-1 

possesses 

K-PS2-2 

Defining 

A  situation  that  people  want  to  change  or  create  can  be 

K-PS2-1 

Engineering 

approached  as  a  problem  to  be  solved  through  engineering. 

K-PS2-2 

Problems 

Such  problems  may  have  many  acceptable  solutions 

Table  3 

NGSS  Performance  Expectations  Related  to  RoboBall 

Related  Science  and  Engineering  Practices 
Asking  Questions  and  Defining  Problems 

Asking  questions  and  defining  problems  in  grades  3-5  builds  on  grades  K-2  experiences  and  progresses  to 
specifying  qualitative  relationships/ 

•  Ask  questions  that  can  be  investigated  based  on  patterns  such  as  cause  and  effect  relationships. 

•  Define  a  simple  problem  that  can  be  solved  through  the  development  of  a  new  or  improved  object 

or  tool. 
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Planning  and  Carrying  out  Investigations 

Kindergarten.  Planning  and  carrying  out  investigations  to  answer  questions  or  test  solutions  to  problems 
in  K-2  builds  on  prior  experiences  and  progresses  to  simple  investigations,  based  on  fair  tests,  which 
provide  data  to  support  explanations  or  design  solutions. 

•  With  guidance,  plan  and  conduct  an  investigation  in  collaboration  with  peers. 

3"^"^  grade.  Planning  and  carrying  out  investigations  to  answer  questions  or  test  solutions  to  problems  in  3- 
5  builds  on  K-2  experiences  and  progresses  to  include  investigations  that  control  variables  and  provide 
evidence  to  support  explanations  or  design  solutions. 

•  Plan  and  conduct  an  investigation  collaboratively  to  produce  data  to  serve  as  the  basis  for  evidence, 

using  fair  tests  in  which  variables  are  controlled  and  the  number  of  trials  are  considered. 

•  Make  observations  and/or  measurements  to  produce  data  to  serve  as  the  basis  for  evidence  for  an 

explanation  of  a  phenomenon  or  test  a  design  solution. 

Analyzing  and  Interpreting  Data 

Analyzing  data  in  K-2  builds  on  prior  experiences  to  collecting,  recording,  and  sharing  observations. 

•  Analyze  data  from  tests  of  an  object  or  tool  to  determine  if  it  works  as  intended. 

In  the  domain  of  mathematics,  RoboBall  is  aligned  with  three  Common  Core  State  Standards: 

1 .  Reason  abstractly  and  quantitatively. 

2.  Describe  measurable  attributes  of  objects,  such  as  length  or  weight.  Describe  several 
measurable  attributes  of  a  single  object. 

3.  Directly  compare  two  objects  with  a  measurable  attribute  in  common,  to  see  which  object 
has  “more  of’  /  “less  of’  the  attribute,  and  describe  the  differences. 

6.2.  RELATED  FORMATIVE  ASSESSMENT  PROBES  (KEELY  &  HARRINGTON, 

2010) 

The  following  are  additional  recommended  considerations  that  the  game  system  could  support. 

o  Is  it  possible  to  make  the  ball  roll  at  a  constant  speed  (i.e.,  when  the  ball  is  neither 
slowing  down  nor  speeding  up)? 

o  What  is  a  force? 

o  Present  students  with  common  misconceptions  about  force  and  motion  and  ask  them  to 
explain  their  thinking  about  those  ideas. 

o  How  would  someone  walk  to  school  if  there  were  no  friction  in  the  world? 

6.3.  RELATED  “REAL-WORLD”  EXAMPLES 

Common  hands-on  exercises  that  can  also  be  used  to  accompany  RoboBall  gameplay  include  the 
following: 

o  Measure  the  distance  and/or  the  time  it  takes  for  various  objects  (cars,  balls,  boxes, 
blocks)  to  travel  along  different  surfaces. 
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o  Have  students  walk  side  by  side  at  different  speeds  while  dropping  a  rock  on  the  ground 
at  steady  intervals.  Examine  the  differences  in  the  ‘rock  trails’  left  by  each  student, 
o  Have  students  generate  examples  of  difference  things  that  exert  a  force  or  are  acted  upon 
by  forces.  Use  those  examples  to  go  on  a  “forces  scavenger  hunt”, 
o  Have  young  children  push  or  pull  on  objects  using  their  own  muscles, 
o  Play  RoboBall  charades  by  giving  students  a  scenario  depicted  in  one  of  the  levels  of 
RoboBall  and  having  kids  act  it  out. 

6.4.  SOCIAL  EMOTIONAL  LEARNING  GOALS  OF  ROBOBALL 

SEE  concepts  covered  in  Roboball  are  emotional  awareness  (i.e.,  emotion  knowledge/recognition) 
and  perspective-taking.  Specific  learning  objectives  would  let  students  recognize  and  label  (i.e., 
identify)  own  and  other’s  emotions  based  on  facial,  nonverbal,  behavioral  and  contextual  cues. 

6.5.  HOW  ROBOBALL  CAN  BE  USED  IN  THE  CLASSROOM 

This  game  can  be  coupled  with  a  math  and  science  curriculum  because  the  SEE  components  of  the 
game  reinforce  students’  ability  to  understand  other  student’  perspectives,  and  thereby  contributes  to 
more  effective  collaboration  and  group  problem-solving  among  students  which  are  foundational 
skills  required  to  effectively  solve  and  understand  many  math  and  science  problems,  especially  when 
working  with  peers. 

6.6.  REAL-WORLD  EXAMPLES  THAT  THIS  GAME  RELATES  TO 

Common  hands-on  SEE  exercises  for  the  classroom  that  relate  to  RoboBall  could  include  prompting 
students  to  engage  in  emotion  recognition  and  perspective-taking  skills  if  disagreements  or  conflicts 
arise  among  peers.  For  instance,  if  two  students  are  stuck  because  they  are  disagreeing  about  how  to 
proceed  with  a  science  problem  or  with  their  approach  to  solving  a  math  problem,  the  students  can 
be  prompted  to  stop  and  assess  how  they  think  the  other  student  is  feeling  and  why. 

Once  students  realize  that  they  are  experiencing  specific  emotions  based  on  their  experience,  and 
that  these  emotions  are  making  it  difficult  for  them  to  continue  with  their  task;  they  can  be  further 
prompted  to  come  up  with  a  solution  or  response  that  will  help  them  feel  better.  The  decision  the 
students  make  with  respect  to  how  to  make  the  other  student  feel  better,  will  help  both  of  the 
students  proceed  with  the  task,  as  they  will  be  taking  the  perspective  of  the  other  student  into 
account  when  making  their  decisions. 
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6.7.  BASIC  GOAL  AND  MECHANIC  OF  THE  GAME: 

The  goal  is  for  the  vector(s)  to  hit  the  ball,  and  then  for  the  ball  to  hit  the  treasure  chest 

Players  must  place  the  force  arrow(s)  (i.e.,  vectors),  ball(s),  and/or  treasure  chest(s)  in  the 
appropriate  location(s)  to  achieve  this  goal 

6.8.  INSTRUCTIONAL  HELP 

Players  will  select  objects  from  the  toolbox  and  place  them  in  the  appropriate  locations  by  clicking 
and  dragging.  Then  player  will  push  the  “play”  button. 


Toolbox 
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Objects  (e.g.,  vectors,  balls,  treasure  chests)  can  only  be  placed  on  locations  this  graphic  marks. 

6.9.  REPRESENTATIONAL  CUES 

In  RoboBall,  our  game  developers  used  icons  and  animations  to  help  convey  underlying  concepts. 

Direction  and  Speed:  Purple  arrows  indicate  that  the  object(s)  will  move  in  that  direction  once  the 
‘play’  button  is  pushed.  They  also  indicate  the  object’s  speed. 


The  ball  cannot  hit  the  comet(s),  but  must  hit  the  treasure  chest. 
Speed:  The  balls  leave  behind  bread  crumb  trails  that  indicate  their  speed: 


Hill  RnliftBall  ii  going  skvwm 
Dots  iro  deier  togetlHf  because  this  object 
is  covering  tod  distance  in  the  hitii  amoyint 
of  time  as  the  object  above. 


Thli  RobciSall  li  going  faifevr; 

Dots  are  ftirliwr  apart  because  this  object  is 
covering  mot*  distance  in  the  him  amount 
of  time  as  the  object  above. 


6.10.  FORCE  MAGNITUDE 


In  this  game,  there  are  two  possible  force  magnitudes: 

{In  introductory  levels,  only  one  fixed  force  magnitude  will  be  present.  However,  in  later  levels,  the 
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player  may  need  to  choose  the  correct  option) 

Small  force  magnitude  -  indicated  by  short  arrow 
Large  force  magnitude  -  indicated  by  long  arrow 

When  the  RoboBalls  are  hit,  they  also  give  off  a  puff  of  smoke  on  impact.  When 
the  same  force  magnitudes  are  used,  the  sizes  of  the  puff  of  smoke  are  the  same. 


ThttM  RoboBalls  w«ra  ghron  tho  same  •!»  foreo: 

Both  of  the  RoboBalls  were  given  big  size  punches  so  the 
'force  puhP  should  be  the  same  size. 


r 

«i 

/  • 

When  you  use  different  force  magnitudes  the  size  of  the  smoke  effects  differ  too. 
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Lar9«  Force  KagnltiKle 

Depicted  by  the  big  puff  of  smoke. 


— smal  Force  Hagnityde 

Depicted  by  the  emai  puff  of  smoke. 


Vectors  may  be  oriented  in  the  following  directions: 

(In  some  levels,  direction  may  be  fixed.  However,  in  some 
levels  the  player  may  need  to  choose  the  correct  option) 


In  this  game,  there  are  two  possible  ball  masses 

(In  introductory  levels,  only  one  fixed  mass  will  be  present. 
However,  in  later  levels,  the  player  may  need  to  choose  the 
correct  option) 

6.11.  A  NEW  SPACEFORCE  VERSION  OF  ROBOBALL 

jb.ll.l.  BRAINPOP  JR.,  REDESIGN  OVERVIEW 

This  section  details  changes  to  RoboBalVs  design  in  response  to  feedback  from  BrainPOP,  Jr. 

This  design  restricted  certain  features  due  to  the  absence  of  persistent  user  IDs.  These  changes  are 
denoted  with  an  asterisk *  *. 

BrainPOP ’s  changes  are  as  follows: 

•  Change  name  of  the  public  game  to  SpaceEorce  in  order  to  distinguish  it  from  the  research 
version  (RoboBall) 

•  Adding  more  in-game  audio 
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•  Add  story  narrative  to  in-game  movies 

•  Added  a  feature  to  make  the  game  movies  skip-able 

•  Remove  SELs  from  the  level  flow  (drive  this  via  a  server  variable?) 

•  Remove  end-of-level  references  to  chest  collection 

•  *Hide  Collections  button  and  screen 

•  *  Hide  Achievements  button  and  screen 

•  *  Unlock  all  levels  in  Mission  Select  menu 

•  Simplify  ”3  strikes  you're  out"  level  flow 

•  Add  "star  rating"  performance  metric  to  end-of-level  menu 

•  *  Make  all  adaptive  levels  mandatory 

•  If  *,  allow  retries  on  assessment  levels;  otherwise,  add  "Captain  Dialog"  to  levels  at  start  of 
unit  and  on  assessment  levels 

•  Add  "progress  indicator"  to  end  level  dialog 

•  Add  "performance  summary"  at  "end  of  unit" 

•  Add  tutorial  for  first  assessment  level 

•  Unlock  the  first  level  in  each  of  unit  in  the  "Review"  menu. 

•  Added  in-game  text  to  further  clarify  the  narrative. 

"LEVEL  END"  SUCCESS  DIALOG  CHANGES 

We  replaced  the  current  "level  end  dialog"  with  a  version  that  rates  the  player's  performance  on  the  Level 

and  shows  their  progress  within  the  Unit. 


The  "Level  Success"  version  of  the  dialog  looks  like  this: 
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When  players  complete  a  level,  the 
game  awards  them  a  score: 

3  stars:  completed  on  first  try 
2  stars:  completed  on  second  try 
1  star:  completed  on  third  try 


When  players  fail  a  level,  they  receive  a 
"strike."  After  three  strikes,  the  player 
must  start  the  unit  over.  Strikes 
accumulate  from  level  to  level. 
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6.12.  CAPTAIN  DIALOG 

At  the  start  of  each  Level,  we  now  present  a  "Captain  Dialog"  that  explains  the  upcoming  challenge 
and  shows  the  player's  progress  through  the  unit: 

The  dialog  will  look  something  like  this: 


The  Captain  Dialog  appears  at  the  start 
of  every  level.  It  shows  the  player's 
progress  through  the  unit,  explains  the 
physics,  and  offers  game  play  hints. 


In  the  first  level  of  each  unit,  the  Captain  Dialog  displays  the  following  text: 


Intro  to  Unit  1:  "Use  the  gloves  to  push  your  RoboBall  into  the  treasures  chests." 

Intro  to  Unit  2:  "The  glove  with  the  big  arrow  pushes  hard.  The  other  glove  pushes  softly." 
Intro  to  Unit  3:  "The  heavier  an  object  is,  the  harder  it  is  to  move  it." 

Intro  to  Chapter  4:  "What  will  happen  when  you  give  an  object  two  pushes  at  once?" 

Intro  to  Chapter  5:  "What  happens  when  you  push  an  object  that  is  already  moving?" 

Intro  to  Chapter  6:  "How  can  you  stop  an  object,  or  make  it  go  backwards?" 

Intro  to  Chapter  7:  "Only  a  few  chests  left,  but  these  are  the  hardest  puzzles." 


6.13.  UNIT  SUMMARY 
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At  the  end  of  each  unit,  the  game  will  provide  a  "unit  summary”  level  which  features  a  "stats” 

dialog: 


The  dialog  will  reports  the  number  of  levels  cleared  and  stars  earned  while  playing  the  unit. 

6.14.  REVISED  LEVEL  ELOW 

The  BrainPOP  team  has  indicated  that  the  existing  level  flow  confuses  players  in  two  ways: 

1)  The  order  of  presentation  of  concepts  seems  muddled,  and 

2)  It’s  unclear  why  the  game  allows  replays  on  some  levels  but  not  others. 

To  combat  these  problems,  we  have  revised  the  level  flow  as  follows: 

1)  Present  assessment  levels  only  once,  at  the  end  of  the  unit  to  which  they  belong. 

2)  Allow  retries  on  all  levels,  including  assessment  levels. 

3)  Make  level  progression  consistent  by  requiring  players  to  complete  all  adaptive  levels*. 

"^Ideally,  we  still  support  adaptive  content  by  placing  "warp  gates"  on  the  progress  track,  then 
allowing  players  to  "warp  past"  adaptive  levels  whose  content  they  had  already  mastered. 
Unfortunately,  this  system  complicates  the  progress  map  and  performance  screens,  so  we 
probably  don't  have  the  time  or  budget  to  implement  it. 

6.15.  DISPLAYING  REMAINING  ATTEMPTS 

We  will  replace  the  existing  "crew”  meter  in  the  upper  right  corner  of  the  level  display  with  a  "strike 
count,”  that  shows  0,  1,  or  2  strikes,  depending  on  the  number  the  player  has  earned. 
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6.16.  NEW  LEARNING  CATEGORIES 

In  the  absence  of  persistent  user  data  (which  we  cannot  collect),  we  track  learning  targets  per  unit. 
We  define  the  new  "per  unit”  targets  in  terms  of  the  learning  targets  used  in  the  original  version  of 
the  game  as  follows: 

Unit  1 :  Which  direction  to  push? 

Unit  2:  How  hard  to  push? 

Unit  3:  Heavy  objects  vs.  light  ones 
Unit  4:  Two  pushes  at  once 
Unit  5:  Adding  pushes 
Unit  6:  Subtracting  pushes 
Unit  7:  Adding  and  subtracting 

6.17.  INITIAL  LEARNING  TARGETS 

We  present  the  learning  targets  in  plain  language,  shown  in  bold  next  to  the  definitions  we  use  in  the 
design  Assessment  and  Adaptivity  Design  document. 

•  M2-1:  The  larger  the  push  or  pull  (impulse)  acting  on  a  body,  the  larger  the  change  in  the  object's 
momentum 

Target  FI:  Students  can  demonstrate  that  a  force  is  a  push  or  pull  that  can  change  an  object’s  position 
and  put  it  into  motion.  BACKGROUND:  The  position  and  motion  of  an  object  can  be  changed  by 
pushing  or  pulling.  An  object  at  rest  will  stay  at  rest  unless  a  force  is  applied. 

•  El-1:  A  stationary  mass,  when  pushed  or  pulled,  moves  in  the  direction  of  the  push  or  pull 

•  El-2:  CLARIEICATION  -  in  the  absence  of  external  forces  like  gravity,  pushes  and  pulls  behave 
identically  in  all  directions 

Target  E2:  Students  will  demonstrate  understanding  that  the  more  massive  an  object,  the  more  force  is 
needed  to  set  the  object  into  motion  in  the  direction  of  the  force.  Mass  is  the  amount  of  matter  in  an 
object.  Mass  is  not  the  same  as  size.  Eorce  is  a  push  or  pull 

•  E2-1:  Given  the  same  impulse,  the  heavier  of  two  masses  will  experience  a  smaller  change  in 
motion 

•  E2-2:  Given  two  objects  of  equal  mass,  that  which  experiences  the  larger  impulse  will  experience 
a  greater  change  in  motion  (this  is  a  restatement  of  M2-1,  so  we  combine  them  -  averaged) 

Target  E3:  Students  are  able  to  explain  that  forces  are  composed  of  magnitude  and  direction. 
BACKGROUND:  Magnitude  of  force  is  its  size  and  strength.  Eorce  always  has  a  direction. 

•  E3-1  (goal  17):  Given  two  impulses  at  right  angles  to  one  another,  students  can  predict  the 
resultant  direction 

•  E3-2  (goal  18):  Given  two  opposing  impulses  acting  on  a  single  mass,  students  correctly  predict 
whether  or  not  impulses  cancel 

6.18.  NEW  LEARNING  CATEGORIES  EXPRESSED  IN  TERMS  OE  OLD 
LEARNING  TARGETS: 

Unit  1:  Which  direction  to  push?  (E 1-1,  El -2) 

Unit  2:  How  hard  to  push?  (All  from  unit  1  and  M2-1) 

Unit  3:  Heavy  objects  vs.  light  ones  (all  from  unit  2  and  E2-1) 

Unit  4:  Two  pushes  at  once  (all  from  unit  3  and  E3-1) 

Unit  5:  Adding  pushes  (all  from  unit  3  and  E3-1) 
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Unit  6:  Subtracting  pushes  (all  from  unit  3  and  F3-2) 

Unit  7:  Adding  and  subtracting  (all  learning  targets) 

6.19.  REVIEW  MENU  CATEGORIES 

We  allow  players  to  jump  to  the  start  of  any  unit  via  the  "Review”  menu.  To  facilitate  this,  we  will 
unlock  the  first  level  of  each  unit  in  the  menu  and  disable  dynamic  unlocking.  This  will  allow  the 
players  to  choose  the  first  level  of  any  unit  at  any  time  in  the  game.  Jumping  to  a  new  unit  resets  the 
player's  accumulated  "stars  and  strikes." 

We’ve  added  an  auto-tutorial  to  the  first  assessment  level  so  students  understand  the  new  placement 
mechanic  (i.e.,  placing  the  chest  instead  of  the  gloves  or  balls). 

6.20.  UIX  IMPROVEMENTS 

We’ve  implemented  the  following  UIX  features: 

•  Improve  the  visibility  of  the  "skip  movie"  arrow 

•  Replace  the  "next  level"  button  with  the  "skip  movie"  arrow 

6.21.  GAME  CONCLUSION 

At  the  conclusion  of  each  Mission  (unit)  the  player  will  receive  a  Mission  Report.  This  report  is 
printable  in  the  event  a  teacher  would  like  proof  of  Mission  completion.  The  Mission  Report 
indicates  the  level  pass/fails  and  the  time  played.  The  player  is  congratulated  and  presented  with  a 
screen  of  their  achievements.  These  achievements,  as  well  as  all  of  the  game  metrics  are  recorded 
by  the  Web  Server. 
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Figure  3.0  -  Student  Friendly  Mission  Report 
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Figure  3.1  -  In-Game  Printer  dialog  for  the  Mission  Report 
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4/23/2014  GAMEUPI  SpaceForce 

Mission  Report 

Crew  Member:  Player  One 
Mission  #:  3 

Topic:  Light  and  Heaw  Objects 
Time:  333 

Sector  1 : 0  misses 
Sector  2:0  misses 
Sector  3:0  misses 
Sector  4:0  misses 
Sector  5: 1  miss 
Sector  6: 2  misses 
Sector  7:2  misses 
Sector  8:0  misses 


hnp://wvMw.brainpop.cani/game6/space(Qrce/  1/1 

Figure  3.2  -  Printed  version  of  the  Mission  Report  for  teacher  verification 


7.  A  MULTI-FACETED  GAME  ENVIRONMENT 


We  have  ultimately  developed  two  versions  of  our  game  available  for  further  learning  by  the  community; 
RoboBall  and  SpaceForce. 

7.1.1.  THE  ROBOBALL  RESEARCH  VERSION 

RoboBall  is  our  research  version  used  mainly  for  efficacy  and  assessment  testing.  It  has  features 
that  support  the  collection  of  finer  user  metrics.  The  OurSpace  game  portal  was  designed 
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specifically  with  data  collection  and  research  in  mind.  It’s  a  one-stop-shop  for  researchers,  game 
designers  and  students.  By  using  authorized  credentials  a  user  can  access  game  data,  redesign 
levels  and  play  the  game.  This  is  possible  because  our  game  is  built  in  html5,  a  powerful  web- 
based  technology. 

When  CRESST  uses  our  RoboBall  game  for  testing  they  know  the  demographics  of  their  student 
population.  We  are  able  to  pre -populate  our  database  with  logins  for  students  in  Kindergarten 
through  Fifth  grade.  This  allows  CRESST  to  analyse  how  well  certain  age  groups  perform.  In 
addition  to  the  pre-populated  logins  the  research  version  includes  adaptive  learning  predicated  on 
integrated  pre  and  post-tests.  When  a  student  reaches  a  pre-test  assessment  we  evaluate  how  well 
they  do  on  the  level  and  record  their  performance.  Then  later  on  in  the  unit  a  post-test  assessment 
is  presented  to  the  user. 

We  typically  present  this  to  the  user  7-10  levels  after  the  pre-test  to  record  their  performance  once 
again.  The  number  of  levels  between  each  assessment  is  determined  by  how  well  the  student 
performs  immediately  following  the  pre-test  at  the  beginning  of  a  unit.  Our  adaptive  learning 
algorithm  calculates  how  well  the  student  is  doing  on  specific  learning  targets  that  the  levels  are 
based  on. 

If  the  student  meets  the  passing  percentage  they  move  ahead  to  more  difficult  levels.  If  the 
student  fails  to  meet  the  minimum  requirement  they  will  receive  remedial  levels  to  reinforce  the 
learning  targets.  These  remedial  levels  restate  the  physics  problem  in  a  different  way  to  give  the 
student  a  new  perspective.  RoboBall,  the  research  version  of  our  game  that  CRESST  uses  for 
testing  can  be  played  here:  http://engage.intific.com/engage  portal/public/. 

7.1.2.  THE  BRAINPOP  SPACEFORCE  VERSION 

The  second  version  of  our  game,  SpaceForce  is  our  public-facing  game  currently  hosted  on 
BrainPOP.  Per  their  recommendations,  we  renamed  RoboBall  to  SpaceForce  to  distinguish  it 
from  the  research  version,  RoboBall. 

In  late  2013  we  reached  out  to  BrainPOP  to  discuss  a  possible  partnership.  We  set  up  a  video 
conference  where  we  discussed  our  RoboBall  game  with  them.  They  wanted  to  know  everything 
about  the  game,  its  learning  strategies  and  goals,  as  well  how  to  play.  We  gave  them  a 
demonstration  to  walk  them  through  all  the  game  has  to  offer.  We  also  provided  technical  and 
creative  information  as  needed. 

BrainPOP  asked  that  we  give  them  access  to  the  game  so  they  could  review  it  with  their  game 
development  team.  We  gave  them  access  to  our  game  and  waited  to  hear  back.  After  reviewing 
the  game  with  their  team  they  informed  us  that  they  were  impressed  with  the  game  and  felt  that 
it’s  learning  goals  match-up  nicely  with  their  audience.  They  gave  us  feedback  on  some  of  the 
areas  that  they  felt  needed  adjustment  based  on  what  their  audience  needs  and  what  their 
technology  supports.  Within  a  few  months  we  revised  the  game  to  meet  their  requirements  and 
got  it  running  on  their  site.  During  this  process  we’ve  built  a  strong  relationship  with  their  team. 
They  asked  us  to  join  them  at  2  education  conferences  in  Austin,  Texas  to  present  our  game  at 
their  booth.  We  accepted  their  invitation  and  presented  at  the  Texas  Computer  Education 
Association  conference  and  at  the  South  by  South  West  Education  conference. 
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SpaceForce  had  several  different  features  added  to  it  based  upon  recommendations  from  the 
BrainPOP  team.  These  were  mainly  due  to  technical  limitations  on  BrainPOP  that  we  had  to 
adjust  for.  BrainPOP  Jr.  doesn’t  support  individual  logins.  This  kept  us  from  leveraging  persistent 
user  data  to  drive  the  progression  based  game  features  like;  individualized  data  capture,  adaptive 
learning,  saved  games,  unlocked  levels,  achievements  and  awards.  Because  of  this  we  created 
new  interfaces  to  communicate  mission  (unit)  selection,  level  progression  and  pass/fail  states. 
SpaceForce  is  available  on  BrainPOP  here:  http://www.brainpop.com/games/spaceforce/. 

7.1.3.  SERVER  AND  GAME  EDITOR  INEORMATION 

The  server  for  OurSpace  is  based  on  a  number  of  commercial  products  (Linux,  Apache,  MySQL, 
PHP:  LAMP  Stack)  and  is  targeted  to  run  in  any  HTML  Browser  that  supports  Elash.  The 
prototype  application  developed  in  Phase  1  was  created  in  Plash.  In  order  to  provide  the  desirable 
level  of  cross  platform  compatibility  and  the  ability  for  multiple  points  of  game  edit/creation 
(Educator,  Developer,  and  Administrator),  we  developed  a  game  engine  in  HTML  5.  This  is  the 
platform  used  to  develop  all  milestones  and  the  final  build  in  Phase  2. 


8.  RECOMMENDATIONS 


Through  this  deliberate  fusion  of  the  most  rewarding  aspects  of  game  creation  and  play,  with  educational 
objectives  and  instructor  lead  learning,  we  have  developed  a  method  and  tool  base  which  can  be  expanded 
to  address  multiple  types  and  levels  of  interactive  learning. 

The  Prototype  Game  is  being  tested  by  CRESST/UCLA,  including  the  statistical  information  being 
collected.  We  anticipate  their  report  by  the  end  of  the  summer  in  order  to  incorporate  the  recommended 
changes  into  the  Test-Ready  game. 

We  have  finished  the  Phase  II  work,  expanding  the  capabilities  with  the  goal  of  interim  deliverables  met 
and  a  test-ready  game  software  after  6  months  and  Pinal  Game  software  23  months  after  award  of  Phase 
II.  The  incremental  funding  allocated  in  the  initial  transaction  was  sufficient  to  continue  work  through 
portions  of  October.  Additional  resources  during  the  early  months  of  Phase  2  let  us  accomplish  the 
engineering,  game  design  and  art  requirements  needed  to  meet  the  test-ready  game  deliverable.  Intific 
recommends  consideration  of  additional  funding,  to  ensure  continuity  of  effort. 

js.l.l.  EXTENDING  OURSPACE  IN  THE  CLASSROOM 

The  concepts  covered  in  RoboBall  can  be  coupled  with  both  science  and  mathematics  curriculum. 
RoboBall  maps  onto  the  Next  Generation  Science  Standards  (NGSS)  (NSP,  2013).  The  game 
mechanics  in  RoboBall  are  consistent  with  the  Performance  Expectations  contained  in  Table  4, 
covers  the  NGSS  Disciplinary  Core  Ideas  contained  in  Table  5,  and  requires  the  use  of  the 
Science  and  Engineering  practices  contained  in  Table  6. 

Table  4 

NGSS  Performance  Expectations  Related  to  RoboBall 

Performance  Expectations 
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K- 

PS2-1 

Plan  and  conduct  an  investigation  to  compare  the  effects  of  different  strengths  or  different 
directions  of  pushes  and  pulls  on  the  motion  of  an  object. 

K- 

PS2-2 

Analyze  data  to  determine  if  a  design  solution  works  as  intended  to  change  the  speed  or 
direction  of  an  object  with  a  push  or  a  pull. 

3-PS2- 

1 

Plan  and  conduct  an  investigation  to  provide  evidence  of  the  effects  of  balanced  and  unbalanced 
forces  on  the  motion  of  an  object. 

MS- 

PS2-2 

Plan  and  conduct  an  investigation  to  provide  evidence  that  the  change  in  an  object’s  motion 
depends  on  the  sum  of  the  forces  on  the  object  and  the  mass  of  the  object. 

Table  5 

NGSS  Disciplinary  Core  Ideas  covered  in  RoboBall 


Disciplinary  Core  Ideas 

Related  Performance 

Expectations 

Forces  and 

Pushes  and  pulls  can  have  different  strengths  and 

K-PS2-1 

Motion 

directions. 

K-PS2-2 

Pushing  and  pulling  on  an  object  can  change  the  speed  or 

K-PS2-1 

direction  of  its  motion  and  can  start  or  stop  it. 

K-PS2-2 

Each  force  acts  on  one  particular  object  and  has  both 
strength  and  a  direction.  An  object  at  rest  typically  has 
multiple  forces  acting  on  it,  but  they  add  to  give  zero  net 
force  on  the  object.  Forces  that  do  not  sum  to  zero  can 
cause  changes  in  the  object’s  speed  or  direction  of  motion. 

3-PS2-1 

The  motion  of  an  object  is  determined  by  the  sum  of  the 
forces  acting  on  it;  if  the  toil  force  on  the  object  is  not  zero, 
its  motion  will  change. 

MS-PS2-2 

Energy 

The  faster  a  given  object  is  moving,  the  more  energy  it 

K-PS2-1 

possesses 

K-PS2-2 

Defining 

A  situation  that  people  want  to  change  or  create  can  be 

K-PS2-1 

Engineering 

approached  as  a  problem  to  be  solved  through  engineering. 

K-PS2-2 

Problems 

Such  problems  may  have  many  acceptable  solutions 

Table  6 
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NGSS  Performance  Expectations  Related  to  RoboBall 

Related  Science  and  Engineering  Practices 
Asking  Questions  and  Defining  Problems 

Asking  questions  and  defining  problems  in  grades  3-5  builds  on  grades  K-2  experiences  and  progresses  to 
specifying  qualitative  relationships/ 

•  Ask  questions  that  can  be  investigated  based  on  patterns  such  as  cause  and  effect  relationships. 

•  Define  a  simple  problem  that  can  be  solved  through  the  development  of  a  new  or  improved  object 
or  tool. 

Planning  and  Carrying  out  Investigations 

Kindergarten.  Planning  and  carrying  out  investigations  to  answer  questions  or  test  solutions  to  problems 
in  K-2  builds  on  prior  experiences  and  progresses  to  simple  investigations,  based  on  fair  tests,  which 
provide  data  to  support  explanations  or  design  solutions. 

•  With  guidance,  plan  and  conduct  an  investigation  in  collaboration  with  peers. 

3"'"^  grade.  Planning  and  carrying  out  investigations  to  answer  questions  or  test  solutions  to  problems  in  3- 
5  builds  on  K-2  experiences  and  progresses  to  include  investigations  that  control  variables  and  provide 
evidence  to  support  explanations  or  design  solutions. 

•  Plan  and  conduct  an  investigation  collaboratively  to  produce  data  to  serve  as  the  basis  for 
evidence,  using  fair  tests  in  which  variables  are  controlled  and  the  number  of  trials  are 
considered. 

•  Make  observations  and/or  measurements  to  produce  data  to  serve  as  the  basis  for  evidence  for  an 
explanation  of  a  phenomenon  or  test  a  design  solution. 

Analyzing  and  Interpreting  Data 

Analyzing  data  in  K-2  builds  on  prior  experiences  to  collecting,  recording,  and  sharing  observations. 

•  Analyze  data  from  tests  of  an  object  or  tool  to  determine  if  it  works  as  intended. 

In  the  domain  of  mathematics,  RoboBall  is  aligned  with  three  Common  Core  State  Standards: 

1 .  Reason  abstractly  and  quantitatively. 

2.  Describe  measurable  attributes  of  objects,  such  as  length  or  weight.  Describe  several 
measurable  attributes  of  a  single  object. 

3.  Directly  compare  two  objects  with  a  measurable  attribute  in  common,  to  see  which  object 
has  ‘‘more  of’  /  “less  of’  the  attribute,  and  describe  the  differences. 

Related  Formative  Assessment  Probes  (Keely  &  Harrington,  2010) 


106 


Data  contained  on  this  page  is  subject  to  restrictions  on  cover  and  notice  page. 


Intific  Inc. 


Alternate  Reality  Teaching:  OurSpace 


N00014-11-C-0593 


•  Is  it  possible  to  make  the  ball  roll  at  a  constant  speed  (i.e.,  when  the  ball  is  neither  slowing  down 
nor  speeding  up)? 

•  What  is  a  force? 

•  Present  students  with  common  misconceptions  about  force  and  motion  and  ask  them  to  explain 
their  thinking  about  those  ideas. 

•  How  would  someone  walk  to  school  if  there  were  no  friction  in  the  world? 

Related  ^^Real-World”  Examples 

Common  hands-on  exercises  that  can  be  used  to  accompany  RoboBall  gameplay  include  the 
following: 

o  Measure  the  distance  and/or  the  time  it  takes  for  various  objects  (cars,  balls,  boxes, 
blocks)  to  travel  along  different  surfaces. 

o  Have  students  walk  side  by  side  at  different  speeds  while  dropping  a  rock  on  the  ground 
at  steady  intervals.  Examine  the  differences  in  the  ‘rock  trails’  left  by  each  student. 

o  Have  students  generate  examples  of  difference  things  that  exert  a  force  or  are  acted  upon 
by  forces.  Use  those  examples  to  go  on  a  “forces  scavenger  hunt”. 

o  Have  young  children  push  or  pull  on  objects  using  their  own  muscles. 

o  Play  RoboBall  charades  by  giving  students  a  scenario  depicted  in  one  of  the  levels  of 
RoboBall  and  having  kids  act  it  out. 


9.  OURSPACE  TEST  RESULTS 


OurSpace  focus  tests  were  conducted  by  CRESST/UCLA  on  the  following  dates: 


•  Summer  2012  (Elash  Build) 

o  Playtested  with  80  students 


o  Pilot  effectiveness  study  with  62  students 

•  February-March  2013  (Beta) 

o  Pilot  effectiveness  study  with  80  students 

•  December  2013  (Beta) 

o  Usability  testing  with  41  students 

•  April  2014  (Pinal) 

o  Usability  testing  20  students  (Most 
finished  the  game) 


o  Grades  and  number  of  students:  3"^^  (3),  4*  (9),  5*  (12). 

Extensive  testing  during  the  summeris  planned,  and  Intific  is  standing  by  to  respond  to  the  results 
obtained. 
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CONTRACT  COSTS 


REPORT  DATE:  05/30/14 

INTIEIC,  Inc. 

PERIOD  ENDING:  05/31/14 

PROJECT  MGR: 

Kristy  Tipton 

JOB  SUMMARY  REPORT 

CLIENT:  ONR 

EST  TOT  VAL: 

3,936,951.00 

PRIME  CONTRACT  ID: 

N00014-11-C-0593 

AS  OE  05/31/14 

EUNDED  VALUE: 

3,936,951.00 

CONTRACT  TYPE 

CPEE 

TOTAL  CONTRACT  INCURRED  COSTS 

Direct  Labor 

1,278,475.00 

TOTAL  LABOR  COST^ 

1.278.475.00 

Subcontractor 

454,454.00 

Direct  Travel 

30,224.00 

Other  ODCs 

18,141.00 

TOTAL  OTHER  DIR.COST 

502,819.00 

TOTAL  INDIRECT  EXP. 

1,897,626.00 

TOTAL  CONTRACT  COSTS 

3,678,919.00 

EEL  COMPUTED  7.00 

257,524.00 

CONTRACT  REVENUE 

3,936,443.00 

^  Note:  this  figure  is  based  on  actuals  through  5/16/2014  and  projections  through  5/30/2014 
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9.1.  HARDWARE  PURCHASES 


Contract  Report:  Hardware  Purchases 

Contract  Number: 

N00014-11-C-0593 

Description: 

ENGAGE  PROGRAM 

Internal  Contact: 

Kristy  Tipton 

Licenses: 

1 

Begin  Date: 

8/10/2011 

End  Date: 

5/31/2014 

Asset  Tag: 

Asset  Type: 

Description: 

Site: 

dENGAGE-10001 

Mac  Mini  desktop 

Mac  Mini  desktop,  Mac  Mini  desktop 

Austin  - 

Jollyville 

dENGAGE-10002 

Samsung  840  SSD  120GB 

Samsung  840  SSD  120GB,  Samsung  840  series  SSD  Hard  Drive  120GB 

Austin  - 

Jollyville 

DIR-10113 

Samsung  840  SSD  120GB 

Samsung  840  SSD  120GB,  Samsung  840  series  SSD  Hard  Drive  120GB 

Austin  - 

Jollyville 

DIR-10113 

Samsung  840  SSD  120GB 

Samsung  840  SSD  120GB,  Samsung  840  series  SSD  Hard  Drive  120GB 

Austin  - 

Jollyville 

INT-11193 

iPad  64GB  Wifi 

iPad  64GB  Wifi,  Refurbished  Gen  1  iPad,  64GB  WiFi 

Austin  - 

Jollyville 
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